Campus bullying behavior detection method and system based on video phone voice recognition

Abstract

The application relates to the technical field of speech recognition, in particular to a campus bullying behavior detection method and system based on video telephone speech recognition. By acquiring sound signals and continuous image frames in a video telephone monitoring area, effective fusion of visual and auditory information is realized, so that abnormal conditions in the campus can be more comprehensively captured and analyzed. The feature points of a person in the image frame are recognized and tracked, and the position distribution between persons is combined to obtain a gathering characteristic value of the person in the monitoring area, which is of great significance to identifying potential bullying scenes. When the gathering characteristic value is abnormal, the sound signals are further analyzed, sound data segments are divided, acoustic characteristics are extracted, and an abnormal factor of each sound data segment is calculated. Finally, the abnormal factors of all sound data segments and the person gathering characteristic value are combined to obtain an abnormal characteristic value for monitoring the campus bullying behavior, so that the risk of missed reports and false reports is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to a campus bullying behavior detection method and system based on video telephone speech recognition. BACKGROUND

[0002] In the field of campus safety, campus bullying has always been the focus of attention of schools, parents and society. Campus bullying behavior causes serious harm to the physical and mental health of the victims, and also destroys the harmonious atmosphere of the campus. By timely discovering and handling campus bullying behavior, schools can better protect the physical and mental health of students and promote the all-round development of students, providing a safer and more harmonious learning environment for students.

[0003] In recent years, with the rapid development of video monitoring technology, more and more schools have begun to use video monitoring to assist in campus safety management. However, pure video monitoring also faces some problems. Video monitoring can capture pictures, but it can only capture abnormal behaviors on the body, and it does not make full use of sound information. Sound often provides important clues, such as arguing, crying and other abnormal sounds, which may be related to campus bullying behavior. Therefore, if only the monitoring pictures are used to judge campus bullying behavior, it may lead to the neglect of verbal bullying and other behaviors, thereby reducing the accuracy and reliability of campus bullying behavior detection. SUMMARY

[0004] In order to solve the technical problem that only relying on monitoring pictures to judge campus bullying behavior will lead to insufficient use of sound information, thereby affecting the accuracy and reliability of campus bullying behavior detection, the purpose of the present application is to provide a campus bullying behavior detection method and system based on video telephone speech recognition, and the technical solution is as follows:

[0005] Obtain the sound signal and continuous multiple image frames in the same period in the video telephone monitoring area;

[0006] Identify the feature points of the characters in the image frames and track the position changes of the feature points in the continuous multiple image frames to quantify the movement feature values of the characters; combine the movement feature values with the position distribution between the characters in all image frames to obtain the gathering feature values between the characters;

[0007] Divide the sound signal according to the data fluctuation in the sound signal to obtain multiple sound data segments; determine the abnormal factor of each sound data segment based on the degree of deviation between the acoustic characteristics represented by each sound data segment and the standard condition;

[0008] The abnormality characteristic value is obtained by integrating the abnormality factors of all sound data segments and the gathering characteristic value between persons in the monitoring area.

[0009] Further, the method for obtaining the moving characteristic value comprises:

[0010] In each adjacent two image frames, feature points of persons in the image frames are recognized and matched, so as to obtain matched feature point pairs;

[0011] In each adjacent two image frames, the Euclidean distance between the positions of two feature points in each matched feature point pair is taken as a displacement factor, and the mean value of the displacement factors between all matched feature point pairs in each adjacent two image frames is taken as a displacement parameter;

[0012] The mean value of all displacement parameters corresponding to continuous multiple image frames is taken as the moving characteristic value.

[0013] Further, the method for obtaining the matched feature point pairs comprises:

[0014] Each image frame is taken as the input of the pre-trained neural network, so as to obtain the person region in each image frame;

[0015] In each image frame, the SIFT algorithm is used to obtain feature points in the person region and the characteristic values corresponding to the feature points;

[0016] Each feature point in each image frame is combined with each feature point in the adjacent next image frame, so as to obtain all feature point pairs, and the Euclidean distance between the characteristic values of two feature points in each feature point pair is taken as a difference index;

[0017] For each feature point in each image, the feature point pair corresponding to the minimum difference index is taken as a matched feature point pair between adjacent two image frames in all feature point pairs in the adjacent next image frame.

[0018] Further, the method for obtaining the gathering characteristic value comprises:

[0019] In each image frame, all person regions are combined two by two, so as to obtain all region combinations;

[0020] In each region combination, the Euclidean distance between the centroids of two person regions is taken as a distance factor, and the product of the mean value of the distance factors corresponding to all region combinations and the minimum value of the distance factors in all region combinations is taken as a distance parameter;

[0021] The ratio of the number of person regions in each image frame to the distance parameter is taken as the gathering factor between persons in each image frame.

[0022] The moving feature value and the aggregation feature value are negatively correlated, and the aggregation factor and the aggregation feature value are positively correlated.

[0023] Further, the method for obtaining the sound data segment comprises:

[0024] Taking the sound signal between every two adjacent extreme points in the sound signal as an initial data segment;

[0025] In each initial data segment, taking the numerical difference between the maximum value and the minimum value as the fluctuation amplitude of each initial data segment;

[0026] In all initial data segments of the sound signal, the initial data segments are merged based on the similarity of the fluctuation amplitudes between adjacent initial data segments, so as to obtain all sound data segments in the sound signal.

[0027] Further, the merging process comprises:

[0028] Taking the absolute value of the difference between the fluctuation amplitudes of the adjacent two initial data segments, after negative correlation mapping and normalization, as a merging index;

[0029] In the sound signal, the first initial data segment in time sequence is taken as a to-be-tested data segment to start traversal, and it is judged whether the merging index between the to-be-tested data segment and the adjacent next initial data segment meets a preset condition, if not, the to-be-tested data segment is taken as a sound data segment, and the next initial data segment adjacent to the to-be-tested data segment is taken as a new to-be-tested data segment to continue traversal;

[0030] If yes, the next initial data segment adjacent to the to-be-tested data segment is incorporated into the to-be-tested data segment to obtain a new to-be-tested data segment, and it is judged whether the merging index between the last initial data segment in the new to-be-tested data segment and the adjacent next initial data segment meets the preset condition, if not, the next initial data segment adjacent to the to-be-tested data segment is taken as a new to-be-tested data segment to continue traversal;

[0031] The preset condition is that the merging index is greater than a preset merging threshold.

[0032] Further, the method for obtaining the abnormality factor comprises:

[0033] In each sound data segment, taking the difference between the mean value of all maximum values and a preset standard value after normalization as a sound amplitude deviation factor;

[0034] The ratio of the number of initial data segments in each sound data segment to the time length of the sound data segment is taken as the fluctuation frequency in each sound data segment, and the normalized value of the difference between the fluctuation frequency and a preset standard frequency is taken as the fluctuation frequency deviation factor;

[0035] The product of the sound amplitude deviation factor and the fluctuation frequency deviation factor of each sound data segment is taken as the anomaly factor of each sound data segment.

[0036] Further, the method for obtaining the anomaly characteristic value comprises:

[0037] The normalized value of the product of the mean value of the anomaly factors of all sound data segments and the gathering characteristic value between persons in the monitoring area is taken as the anomaly characteristic value.

[0038] Further, the method for obtaining the detection result comprises:

[0039] When the anomaly characteristic value is greater than or equal to a preset anomaly threshold value, it is judged that bullying phenomenon occurs, and a warning is given; when the anomaly characteristic value is less than the preset anomaly threshold value, it is judged that bullying phenomenon does not occur.

[0040] A campus bullying behavior detection system based on video telephone voice recognition comprises a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set.

[0041] The present application has the following beneficial effects:

[0042] In view of the problem that the prior art lacks utilization of sound information when detecting campus bullying behavior, the application obtains sound signals and continuous multiple image frames in the same period in a video intercom monitoring area, so that effective fusion of visual information and auditory information can be realized, and abnormal conditions in the campus can be captured and analyzed more comprehensively. Since there is often body contact or personnel gathering when campus bullying behavior occurs, the feature points of the characters in the image frames are recognized, and the position changes of the feature points in the continuous image frames are tracked, and the position distribution of the characters in the image frames is combined to obtain the gathering feature value between the characters in the monitoring area, which is of great significance for identifying potential bullying scenes. It is not enough to accurately evaluate whether bullying behavior has occurred only according to the gathering feature value, so the sound signals are further analyzed. The sound signals are divided according to the data fluctuation to obtain sound data segments, each of which can represent a continuous sound. At the same time, since there are crying, quarreling and other situations when bullying occurs, which are often accompanied by high-pitched and fast speech, etc., the deviation between the acoustic characteristics of each sound data segment and the standard situation is obtained based on the acoustic characteristics of each sound data segment to obtain the abnormal factor of each sound data segment, which provides a strong basis for the judgment of bullying behavior. Finally, the abnormal feature value is obtained by comprehensively considering the abnormal factor of all sound data segments and the gathering feature value between the characters in the monitoring area. This comprehensive evaluation process can comprehensively reflect the abnormal conditions in the monitoring area, reduce the risk of false positives and false negatives, and provide more comprehensive and effective protection for campus safety. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 The method flowchart of a campus bullying behavior detection method based on video intercom voice recognition provided by an embodiment of the present application;

[0045] Figure 2 The method flowchart of a mobile feature value acquisition method provided by an embodiment of the present application;

[0046] Figure 3 The system block diagram of a campus bullying behavior detection system based on video intercom voice recognition provided by an embodiment of the present application;

[0047] Figure 4A system structure schematic diagram of a campus bullying behavior detection system based on video phone voice recognition provided by one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined object, the following describes in detail the specific implementation, structure, features and effects of a campus bullying behavior detection method and system based on video phone voice recognition according to the present application, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0050] The following specifically describes the specific scheme of a campus bullying behavior detection method and system based on video phone voice recognition provided by the present application, in combination with the accompanying drawings.

[0051] Please refer to Figure 1 which shows a method flowchart of a campus bullying behavior detection method based on video phone voice recognition provided by one embodiment of the present application. The method includes the following steps:

[0052] Step S1: Acquire sound signals and continuous multiple image frames in the same period in the video phone monitoring area.

[0053] In the vital field of campus safety, the problem of campus bullying has long been the focus of attention of school management, parents and the community. Campus bullying behavior not only causes immeasurable harm to the physical and mental health of the victims, but also seriously damages the harmony and tranquility of the campus. Therefore, timely detection and proper handling of campus bullying behavior is an important part of protecting students' physical and mental health, promoting students' overall development, and creating a safe and harmonious learning environment for schools.

[0054] In recent years, more and more schools have begun to use video monitoring technology to strengthen campus safety management. However, video monitoring mainly focuses on abnormal behavior of the body, and underutilizes sound, which is also an important source of information. In fact, sound often provides key clues, such as abnormal sounds like arguing and crying, which may be closely related to campus bullying behavior.

[0055] Therefore, in the embodiment of the present application, visual information is combined with auditory information, specifically, a video phone monitoring device with audio and video acquisition function is installed in the key areas of the campus (corridor, playground, dining hall, staircase), to ensure that the video phone can cover the key areas and the quality, resolution and audio acquisition capability of the video phone can meet the needs of subsequent analysis. Then set the monitoring time period, and use the audio acquisition module built-in the video phone to capture the sound signal in the monitoring area, and the sound signal is a time sequence signal with two dimensions of time and amplitude. At the same time, set a suitable frame rate, and use the camera part of the video phone to collect continuous multiple image frames in the monitoring area under the same monitoring time period.

[0056] It should be noted that the length of the monitoring time period can be set to 3 minutes, wherein the sampling frequency of the sound signal is set to 240HZ, and the image frame can be collected every 5s. The specific sampling frequency and the acquisition time interval of the image frame can be adjusted according to the implementation scene, which is not limited here; at the same time, the image frame and the sound signal can also be denoised respectively to enhance the image contrast and remove the interference of environmental noise in the sound signal. The sound signal can also be pre-emphasized to enhance the high frequency components in the sound signal. The denoising process and the pre-emphasis process are both known technologies, which will not be described here.

[0057] Step S2: identifying the feature points of the person in the image frame and tracking the position change of the feature points in the continuous multiple image frames to quantify the movement characteristic value of the person; combining the movement characteristic value with the position distribution between the persons in all image frames to obtain the gathering characteristic value between the persons.

[0058] In the process of campus bullying, the distance between the attacker and the victim is relatively close, and the position will not move greatly in a period of time, therefore, the image frames in the monitoring area can be analyzed, the feature points of the person are identified and the position change of the feature points in the continuous image frames is tracked, so as to reflect the movement of the person in the monitoring area and obtain the movement characteristic value as an important index to evaluate the potential bullying scene. At the same time, since the campus bullying usually appears the gathering of personnel, the monitoring area will also appear the gathering of personnel, therefore, the movement characteristic value is combined with the position distribution between the persons in all image frames to obtain the gathering characteristic value between the persons in the monitoring area. The gathering characteristic value analyzes the gathering degree of the persons in the monitoring area from two dimensions of the gathering of the persons and the movement of the persons, which is helpful to more accurately identify the potential bullying scene.

[0059] Tracking of feature points is crucial for recognizing the behavior pattern of a person, and can effectively distinguish individuals in a complex background or a crowded situation, so as to more accurately identify the action trajectory of a person, and help improve the accuracy of campus bullying scene recognition. Therefore, the feature points of a person in an image frame are first identified and tracked in the position change of the feature points in continuous multiple image frames, and the movement feature value of the person in the image frame is quantified.

[0060] Preferably, in an embodiment of the present application, the method for obtaining the movement feature value comprises:

[0061] Referring to Figure 2 , a method flowchart of a method for obtaining a movement feature value in an embodiment of the present application is shown, and the method comprises the following steps:

[0062] Step S201: In each adjacent two image frames, the feature points of a person in the image frame are identified and matched, so as to obtain a matched feature point pair.

[0063] Each image frame is taken as an input of a pre-trained neural network, and the neural network can automatically identify and mark the person region in the image, so that the person region in each image frame can be obtained.

[0064] Then, since the Scale-Invariant-Feature-Transform (SIFT) algorithm can detect local features in the image frame, the SIFT algorithm is used to extract feature points in the person region in each image frame. At the same time, for each feature point, the local gradient of the image in the neighborhood around the feature point is measured, and the gradient information is converted into a feature vector, that is, the feature value (descriptor) corresponding to each feature point is obtained. Each feature point in each image frame is combined with each feature point in the adjacent next image frame to obtain all feature point pairs. At this time, each feature point in each image frame corresponds to multiple feature point pairs (between the adjacent next image frame).

[0065] Next, the Euclidean distance between the feature values of the two feature points in each feature point pair is calculated as a difference index. The greater the Euclidean distance, the greater the difference index, which represents that the difference between the two feature points in the feature space is greater. Conversely, the smaller the difference index, which means that the two feature points may be the same feature point in different image frames.

[0066] Finally, for the convenience of explanation and illustration, in each image, optionally one feature point is taken as a target point, and for the target point, in all its corresponding feature point pairs (between adjacent next image frames), the feature point pair with the minimum difference index is taken as a matching feature point pair between the adjacent two image frames, and at this time, the two feature points in the matching feature point pair in the two image frames are considered to represent the same feature.

[0067] It should be noted that the training process of the neural network is a process known to those skilled in the art, which will not be repeated here; at the same time, the SIFT algorithm is also a known technology, which will not be repeated here.

[0068] Step S202: In adjacent image frames, based on the positional relationship between the two feature points in the matching feature point pair, the displacement parameter of the person moving in the two image frames is determined.

[0069] Through the feature point matching in step S201, the position change of the same person can be tracked across different image frames. Therefore, in each adjacent two image frames, the Euclidean distance between the positions of the two feature points in each matching feature point pair is taken as a displacement factor. The greater the displacement factor, the greater the change in the position of the feature point, and the more obvious the moving feature of the corresponding person. Finally, the average value of the displacement factors between all matching feature point pairs in the adjacent two image frames is taken as the displacement parameter. At this time, the greater the displacement parameter, the greater the amplitude of the person moving in the adjacent two image frames, and the lower the possibility of representing a potential bullying scene.

[0070] It should be noted that when calculating the displacement factor, the same coordinate system can be constructed for all image frames, for example, taking the lower left corner of the image frame as the origin, the horizontal right direction as the horizontal axis, and the vertical upward direction as the vertical axis. At this time, each feature point has a two-dimensional coordinate in its own image frame, and the displacement factor can be calculated based on this.

[0071] Step S203: The displacement parameters of the person moving between all image frames are integrated to obtain the moving feature value of the person in the image frames.

[0072] In the foregoing step, each pair of matching feature points between each two adjacent image frames has a displacement parameter, which represents the movement of the person in the two adjacent image frames Therefore, the mean value of all displacement parameters corresponding to the plurality of continuous image frames is taken as a movement feature value, denoted as YT The movement feature value can represent the movement of all persons in the image frame in the monitoring time period, and the greater the movement feature value, the greater the movement amplitude of all persons in the image frame, which means that the persons are in normal social activities, and the possibility of the bullying scene in the monitoring area is lower; on the contrary, the smaller the movement feature value, the smaller the movement amplitude of all persons in the image frame, which implies a kind of static or concentrated behavior, and the possibility of the bullying scene in the monitoring area is higher.

[0073] The movement feature value reflects the overall movement of the persons in the monitoring area, can capture the activity pattern and dynamic change of the persons, and thus can be used as an index for evaluating whether the bullying scene exists; and the position distribution between the persons in each image frame provides static spatial information, explains the relative position and gathering state of the persons in space, and the combination of the two types of information can comprehensively consider the activity of the persons in the monitoring area, obtain the gathering feature value, and thus facilitate more accurate judgment of whether the potential bullying problem exists.

[0074] Preferably, in an embodiment of the present application, the method for obtaining the gathering feature value comprises:

[0075] In each image frame, all person regions are combined two by two to obtain all region combinations.

[0076] Then in each region combination, the Euclidean distance between the centroids of the two person regions is calculated as a distance factor, and the smaller the distance factor, the closer the distance between the two person regions; the product of the mean value of the distance factors corresponding to all region combinations and the minimum value of the distance factors in all region combinations is taken as a distance parameter, and the smaller the distance parameter, the smaller the distance between all persons in each image frame, that is, the higher the gathering degree, which is more likely to exceed the normal social distance, and implies the existence of the bullying scene.

[0077] Since the bullying process occurs while the characters gather, the phenomenon of gathering a crowd also occurs, and the degree of gathering a crowd can be represented by the number of character regions, so according to the distance parameter and the number of character regions, the gathering factor between characters in each image frame is calculated, and the more the number of character regions, the more the phenomenon of gathering a crowd occurs, and if the distance parameter is smaller, it means that the phenomenon of character gathering occurs, and at this time, the possibility of bullying scene is higher. It should be noted that in this embodiment of the present application, it is considered that the centroids of different character regions in the same image frame do not coincide, that is, the value of the distance parameter is not 0. Therefore, in this embodiment of the present application, the ratio of the number of character regions in each image frame to the distance parameter is taken as the gathering factor between characters in each image frame, denoted as JY, at this time, the larger the gathering factor, the higher the possibility of bullying scene.

[0078] Finally, the gathering feature value is determined by synthesizing the moving feature value and the gathering factor of all image frames, and the formula model of the gathering feature value includes:

[0079]

[0080] Wherein, JT represents the gathering feature value; n represents the number of image frames; JY i represents the gathering factor of characters in the ith image frame; YT represents the moving feature value; γ1 represents a preset first parameter; norm() represents a normalization function.

[0081] In the formula model of the gathering feature value, based on the foregoing analysis, the larger the gathering factor, the higher the possibility of bullying scene in each image frame; at the same time, the smaller the moving feature value, the smaller the moving amplitude of all characters in the image frame, which implies a certain static or concentrated behavior, and the higher the possibility of bullying scene in the monitoring area, so the mean value of the gathering factors of all image frames is taken as the numerator part, and the moving feature value is taken as the denominator part, so as to realize the logical relationship described above, and the gathering feature value is obtained after the ratio is normalized, at this time, the larger the gathering feature value, the higher the possibility of bullying behavior in the monitoring time period.

[0082] It should be noted that the function of the preset first parameter γ1 is to prevent the denominator from being 0, and the value can be 0.001, and the specific value can be adjusted according to the implementation scene, which is not limited here.

[0083] At this point, by identifying and analyzing the images of the video phone, the possibility of campus bullying is preliminarily determined.

[0084] Step S3: dividing the sound signal according to the data fluctuation in the sound signal to obtain a plurality of sound data segments; determining an abnormality factor of each sound data segment based on a deviation degree between an acoustic characteristic represented by each sound data segment and a standard condition.

[0085] The aggregation characteristic value indicates the aggregation of the person in the monitoring picture, and when the campus bullying occurs, the aggressor usually performs verbal humiliation, the tone is pulled high, the speaking speed is fast, and the victim's crying and the quarrel between the two parties are also possible, which are all features that can be analyzed and quantified on the sound signal, and are helpful to more accurately determine whether the bullying phenomenon occurs in the monitoring time period.

[0086] The sound data segments are obtained by dividing the sound signal according to the data fluctuation in the sound signal, each sound data segment can represent a continuous sound at this time, the sound features in each sound data segment are more concentrated and consistent, which is helpful to improve the accuracy of subsequent processing.

[0087] Preferably, in an embodiment of the present application, the method for obtaining the sound data segment comprises:

[0088] The extreme points in the sound signal are obtained and the sound signal is initially divided according to the extreme points, that is, the sound signal between each adjacent two extreme points is taken as an initial data segment, each initial data segment obtained at this time represents a fluctuation. It should be noted that the method for obtaining the extreme points is a known technology, which will not be described here.

[0089] Then in each initial data segment, the numerical difference between the maximum value and the minimum value is taken as the fluctuation amplitude of each initial data segment, the greater the fluctuation amplitude, the greater the fluctuation intensity of the sound.

[0090] In view of the fact that each initial data segment represents a fluctuation, and in a real scene, a continuous sound with consistent characteristics can be represented by multiple fluctuations on a sound signal, in all initial data segments of the sound signal, the similarity between the fluctuation amplitudes of each adjacent two initial data segments is analyzed to obtain a merging index, so as to merge the initial data segments according to the merging index, and obtain all sound data segments in the sound signal. At this time, each sound data segment can more accurately represent the characteristics of a continuous sound. Specifically, the absolute value of the difference between the two fluctuation amplitudes can be calculated. The greater the absolute value of the difference, the greater the difference between the fluctuation amplitudes of the adjacent two initial data segments, the worse the consistency of the sound characteristics, and the lower the possibility of merging. Therefore, the value obtained by negatively correlating and normalizing the absolute value of the difference is used as the merging index to correct the logical relationship. At this time, the greater the merging index, the more the two initial data segments should be merged. It should be noted that the negatively correlated and normalized method can use an exp(-x) function, where exp represents an exponential function with a natural constant e as the base, and x represents the independent variable.

[0091] Finally, in the sound signal, the first initial data segment in time sequence is taken as a to-be-tested data segment to start traversal, and it is judged whether the merging index between the to-be-tested data segment and the adjacent next initial data segment meets a preset condition. If not, the to-be-tested data segment is taken as a sound data segment, and the next initial data segment adjacent to the to-be-tested data segment is taken as a new to-be-tested data segment to continue traversal.

[0092] If yes, the next initial data segment adjacent to the to-be-tested data segment is merged into the to-be-tested data segment to obtain a new to-be-tested data segment, and it is judged whether the merging index between the last initial data segment in the new to-be-tested data segment and the adjacent next initial data segment meets the preset condition. If not, the next initial data segment adjacent to the to-be-tested data segment is taken as a new to-be-tested data segment to continue traversal.

[0093] After the traversal is completed, all sound data segments can be obtained.

[0094] The preset condition is that the merging index is greater than a preset merging threshold. It should be noted that the preset merging threshold is set to 0.5, and the specific value can be adjusted according to the implementation scene, which is not limited herein.

[0095] The above merging process is illustrated as follows: for example, there are 5 initial data segments in a sound signal, and the 5 initial data segments are sequentially arranged as initial data segment 1, initial data segment 2, initial data segment 3, initial data segment 4, and initial data segment 5. In the initial merging stage, the initial data segment 1 is taken as a to-be-tested data segment, and it is determined whether the merging index between the to-be-tested data segment and the initial data segment 1 is greater than the merging threshold. If yes, the initial data segment 1 and the initial data segment 2 are merged, and the merged data segment is taken as a new to-be-tested data segment. At this time, it is continuously determined whether the merging index between the initial data segment 2 in the new to-be-tested data segment and the initial data segment 3 is greater than the merging threshold. If no, the data segment obtained by merging the initial data segment 1 and the initial data segment 2 is taken as a sound data segment, and the initial data segment 3 is taken as a new to-be-tested data segment. It is continuously determined whether the merging index between the initial data segment 3 and the initial data segment 4 is greater than the merging threshold. If no, the initial data segment 3 is directly taken as a sound data segment, and at this time, the initial data segment 4 is taken as a new to-be-tested data segment, and it is determined whether the merging index between the remaining initial data segment 5 and the initial data segment 4 is greater than the merging threshold. If yes, the initial data segment 4 and the initial data segment 5 are merged to obtain a new sound data segment.

[0096] When there is campus bullying, the attacker usually speaks in a high-pitched tone and at a fast speed, and may also be accompanied by the victim's crying and the quarrel between the two parties. These acoustic characteristics can be manifested as large amplitude and high fluctuation frequency in the sound signal. Therefore, by quantifying the deviation degree between the acoustic characteristics of each sound data segment and the standard situation, an anomaly factor of each sound data segment can be obtained as another indicator for measuring the possibility of campus bullying in the monitoring time period.

[0097] Preferably, in an embodiment of the present application, the method for obtaining the anomaly factor comprises:

[0098] Since in the campus bullying scene, the sound usually shows a high-pitched tone and a fast speed, both of which can be characterized by the amplitude and fluctuation frequency of the sound signal, in the embodiment of the present application, the acoustic characteristics mainly analyze the amplitude and fluctuation frequency. In each sound data segment, the normalized value of the difference between the mean value of all maximum values and the preset standard value is taken as the sound amplitude deviation factor. The preset standard value can represent the intensity or amplitude of the sound signal in the normal situation. Therefore, the larger the difference between the mean value of the maximum value and the preset standard value, the larger the sound amplitude deviation factor, which indicates that the tone of the sound signal in the current monitoring area is likely to be too high, and the possibility of bullying phenomenon is also significantly increased. It should be noted that the difference here can be positive or negative, so the normalization method here adopts the Sigmoid() function.

[0099] Since each initial data segment represents a data fluctuation in the sound signal, a ratio of a number of initial data segments in each sound data segment to a time length of the sound data segment is taken as a fluctuation frequency in each sound data segment, the fluctuation frequency represents a number of times of sound wave vibration, the greater the fluctuation frequency, the faster the speaking speed of the speaker; then a normalized value of a difference between the fluctuation frequency and a preset standard frequency is taken as a fluctuation frequency deviation factor, wherein the preset standard frequency can represent a frequency of sound wave vibration in the sound signal in a normal scenario, so the greater the difference between the fluctuation frequency and the preset standard frequency, the greater the fluctuation frequency deviation factor, which indicates that the speaking speed of the sound signal in the current monitoring area can be too fast, leading to frequent vibration of the sound signal, and also indicating that the possibility of bullying phenomenon occurring is significantly increased. It should be noted that the difference can be positive or negative, so the normalization method here also uses the Sigmoid() function.

[0100] Based on the foregoing analysis, the greater the sound amplitude deviation factor, the greater the possibility of bullying phenomenon occurring; the greater the fluctuation frequency deviation factor, the greater the possibility of bullying phenomenon occurring, so the sound amplitude deviation factor and the fluctuation frequency deviation factor are positively correlated with the possibility of bullying phenomenon occurring in the monitoring area, so in this embodiment of the present application, a product of the sound amplitude deviation factor and the fluctuation frequency deviation factor of each sound data segment is taken as an abnormal factor of each sound data segment, at this time the greater the abnormal factor, the higher the possibility of bullying phenomenon occurring in the monitoring area under the auditory information.

[0101] It should be noted that the preset standard value and the preset standard frequency can be obtained by: using the video telephone to record a sound signal in the monitoring area in a normal situation in advance, so as to calculate a mean value of all maximum values in the sound signal as the preset standard value, taking the sound signal between adjacent extreme value points as a data segment, calculating a ratio of a number of data segments to a time length of the sound signal as the preset standard frequency, and the specific values of the preset standard value and the preset standard frequency can be adjusted according to the implementation scene, which is not limited here.

[0102] At this point, the possibility of bullying in the campus is further determined by recognizing and analyzing the voice information of the video telephone.

[0103] Step S4: integrate the abnormal factors of all sound data segments and the gathering characteristic values between the persons in the monitoring area to obtain an abnormal characteristic value; according to the abnormal characteristic value, the bullying behavior in the campus is detected to obtain a detection result.

[0104] In the foregoing process, two indexes of judging whether the campus bullying occurs are calculated from the visual angle and the auditory angle respectively. In this step, the two indexes are combined to obtain an abnormal characteristic value, so as to more comprehensively measure the possibility of the campus bullying behavior in the monitoring area, and obtain the monitoring result of the campus bullying behavior.

[0105] Preferably, in an embodiment of the present application, the method for obtaining the abnormal characteristic value comprises:

[0106] Based on the analysis in step S2, the greater the gathering characteristic value is, the more people in the monitoring area and the smaller the moving range of all the people in the image frame, which implies a certain static or concentrated behavior, and the higher the possibility of the bullying scene in the monitoring area is. Meanwhile, based on the analysis in step S3, the greater the abnormal factor is, the higher the speaking tone of the people represented by the sound signal in the monitoring area is, and the faster the speaking speed of the speaker is, and the higher the possibility of the bullying phenomenon is.

[0107] Therefore, the gathering characteristic value and the abnormal factor are positively correlated with the possibility of the bullying phenomenon in the monitoring area, and thus in this embodiment of the present application, the product of the mean value of the abnormal factor of all the sound data segments and the gathering characteristic value between the people in the monitoring area is normalized as the abnormal characteristic value in the monitoring area. When the abnormal characteristic value is greater, the possibility of the bullying in the monitoring area is higher. The normalization is a technique well known to those skilled in the art, and the selection of the normalization function can be linear normalization or standard normalization, and the specific normalization method is not limited herein.

[0108] After obtaining the index for measuring the possibility of the bullying, the index can be used for judgment, so as to realize the detection of the campus bullying behavior and obtain the detection result.

[0109] Preferably, in an embodiment of the present application, the method for obtaining the detection result comprises:

[0110] When the abnormal characteristic value is greater than or equal to the preset abnormal threshold, it is judged that the bullying phenomenon occurs, and a warning is given, and the telephone of the contact person or the manager can be dialed to facilitate the contact person or the manager to timely handle the campus bullying phenomenon. When the abnormal characteristic value is less than the preset abnormal threshold, it is judged that the bullying phenomenon does not occur, and no warning is given.

[0111] It should be noted that the preset abnormal threshold is 0.65, and the specific value can be adjusted according to the implementation scene, which is not limited herein.

[0112] To sum up, in view of the problem that the prior art lacks utilization of sound information when detecting campus bullying behavior, the embodiment of the present application acquires sound signals and continuous image frames in the same period in the video intercom monitoring area, so that effective fusion of visual information and auditory information can be realized, and abnormal conditions in the campus can be captured and analyzed more comprehensively. Since there is often body contact or personnel gathering when campus bullying behavior occurs, the feature points of the characters in the image frames are recognized, and the position changes of the feature points in the continuous image frames are tracked, and the position distribution of the characters in the image frames is combined to obtain the gathering feature value between the characters in the monitoring area, which is of great significance for identifying potential bullying scenes. It is insufficient to accurately evaluate whether bullying behavior occurs only according to the gathering feature value, so the sound signals are further analyzed. The sound data segments are obtained by dividing the sound signals according to the data fluctuation, at this time each sound data segment can represent a continuous sound, and since there are crying, quarreling and other situations when bullying occurs, which are often accompanied by high-pitched and fast speech, etc., the deviation between the acoustic characteristics of each sound data segment and the standard situation is obtained to obtain the abnormal factor of each sound data segment, which provides a strong basis for the judgment of bullying behavior. Finally, the abnormal feature value is obtained by comprehensively considering the abnormal factor of all sound data segments and the gathering feature value between the characters in the monitoring area, and the comprehensive evaluation process can comprehensively reflect the abnormal conditions in the monitoring area, and reduce the risk of false negatives and false positives, thereby providing more comprehensive and effective protection for campus safety.

[0113] The embodiment of the present application also provides a campus bullying behavior detection system based on video intercom voice recognition, please refer to Figure 3 which shows a system block diagram of a campus bullying behavior detection system based on video intercom voice recognition, comprising: a data acquisition module 301 for realizing step S1 in the above method; a character feature analysis module 302 for realizing step S2 in the above method; a sound abnormal feature analysis module 303 for realizing step S3 in the above method; and a campus bullying behavior detection module 304 for realizing step S4 in the above method.

[0114] It should be noted that the system provided in the above embodiment is only exemplified by the division of the above functional modules, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above described functions. In addition, the campus bullying behavior detection system based on video intercom voice recognition and the campus bullying behavior detection method based on video intercom voice recognition provided in the above embodiment belong to the same concept, and the specific implementation process is described in detail in the method embodiment, which will not be repeated here.

[0115] Referring to Figure 4 Fig. 1 shows a system structure diagram of a campus bullying behavior detection system based on video phone voice recognition according to an embodiment of the present application, which comprises a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected through the bus 402; wherein the memory 401 can contain a high-speed random access memory, the bus 402 can be an ISA bus, a PCI bus or an EISA bus, etc., the processor 400 can be an integrated circuit chip with signal processing capability; the memory 401 stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize steps of a campus bullying behavior detection method based on video phone voice recognition.

[0116] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0117] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments.

Claims

1. A method for detecting school bullying behavior based on videophone speech recognition, characterized in that, The method includes: Acquire audio signals and multiple consecutive image frames within the video phone monitoring area during the same period; Feature points of people in image frames are identified and their positional changes are tracked across multiple consecutive image frames. The movement feature values ​​of the people are quantified, representing the magnitude of their movement within the image frames. These movement feature values ​​are then combined with the positional distribution of people across all image frames to obtain clustering feature values. The positional distribution represents the relative spatial positions and clustering states of the people. Clustering feature values ​​are determined based on the movement feature values ​​and the clustering factors of all image frames. The clustering factor for each image frame is the ratio of the number of person regions in that frame to a distance parameter. The distance parameter is the product of the mean of the distance factors corresponding to all region combinations and the minimum distance factor among all region combinations. The distance factor corresponding to a region combination is the Euclidean distance between the centroids of the two person regions within that combination. All region combinations are obtained by pairwise combining all person regions in each image frame. Movement feature values ​​are negatively correlated with clustering feature values, while clustering factors are positively correlated with clustering feature values. The sound signal is divided into multiple sound data segments based on the data fluctuations in the sound signal; based on the degree of deviation between the acoustic characteristics exhibited by each sound data segment and the standard condition, the anomaly factor of each sound data segment is determined. By combining the anomaly factors of all audio data segments and the clustering feature values ​​between people in the monitored area, anomaly feature values ​​are obtained; based on the anomaly feature values, school bullying behavior is detected, and the detection results are obtained. The method for obtaining motion feature values ​​includes: in every two adjacent image frames, identifying feature points of the person in the image frame and matching the feature points to obtain matching feature point pairs; in every two adjacent image frames, using the Euclidean distance between the positions of the two feature points in each matching feature point pair as a displacement factor, and using the average of the displacement factors between all matching feature point pairs in every two adjacent image frames as a displacement parameter; and using the average of all displacement parameters corresponding to multiple consecutive image frames as motion feature values.

2. The method for detecting school bullying behavior based on videophone speech recognition according to claim 1, characterized in that, Methods for obtaining matching feature point pairs include: Each image frame is used as input to a pre-trained neural network to obtain the human figure region in each image frame. In each image frame, the SIFT algorithm is used to obtain the feature points in the human figure region and the feature values ​​corresponding to the feature points; Each feature point in each image frame is combined with each feature point in the next adjacent image frame to obtain all feature point pairs. The Euclidean distance between the feature values ​​of the two feature points in each feature point pair is calculated as a difference index. For each feature point in each image, among all feature point pairs in the next adjacent image frame, the feature point pair corresponding to the smallest difference index is taken as a matching feature point pair between the two adjacent image frames.

3. The method for detecting school bullying behavior based on videophone speech recognition according to claim 1, characterized in that, Methods for acquiring audio data segments include: Take the sound signal between every two adjacent extreme points in the sound signal as an initial data segment; In each initial data segment, the numerical difference between the maximum and minimum values ​​is used as the fluctuation range of each initial data segment; In all the initial data segments of the sound signal, the initial data segments are merged based on the similarity of the fluctuation amplitude between adjacent initial data segments, thereby obtaining all the sound data segments in the sound signal.

4. The method for detecting school bullying behavior based on videophone speech recognition according to claim 3, characterized in that, The merger process includes: The absolute value of the difference between the fluctuation amplitudes of two adjacent initial data segments is negatively correlated and normalized, and then used as the merged index. In the audio signal, the first initial data segment in the time sequence is taken as the data segment to be tested and the traversal begins. It is determined whether the merging index between the data segment to be tested and the next adjacent initial data segment meets the preset condition. If it does not meet the condition, the data segment to be tested is taken as an audio data segment, and the next adjacent initial data segment is taken as a new data segment to be tested and the traversal continues. If the conditions are met, the next initial data segment adjacent to the data segment to be tested will be merged into the data segment to be tested to obtain a new data segment to be tested. It is then determined whether the merging index between the last initial data segment and the next adjacent initial data segment in the new data segment to be tested meets the preset conditions. If not, the next adjacent initial data segment will be used as the new data segment to be tested and the traversal will continue. The preset condition is that the merging index is greater than the preset merging threshold.

5. The method for detecting school bullying behavior based on videophone speech recognition according to claim 3, characterized in that, Methods for obtaining abnormal factors include: In each audio data segment, the normalized value of the difference between the mean of all maximum values ​​and the preset standard value is used as the audio amplitude deviation factor. The ratio of the number of initial data segments to the duration of each audio data segment is used as the fluctuation frequency in each audio data segment, and the normalized value of the difference between the fluctuation frequency and the preset standard frequency is used as the fluctuation frequency deviation factor. The product of the sound amplitude deviation factor and the fluctuation frequency deviation factor for each sound data segment is used as the anomaly factor for each sound data segment.

6. The method for detecting school bullying behavior based on videophone speech recognition according to claim 1, characterized in that, Methods for obtaining abnormal feature values ​​include: The normalized value obtained by multiplying the mean of the anomaly factors of all audio data segments with the cluster feature values ​​among people in the monitored area is used as the anomaly feature value.

7. The method for detecting school bullying behavior based on videophone speech recognition according to claim 1, characterized in that, Methods for obtaining test results include: When the abnormal feature value is greater than or equal to the preset abnormal threshold, it is determined that bullying has occurred and an early warning is issued; when the abnormal feature value is less than the preset abnormal threshold, it is determined that no bullying has occurred.

8. A school bullying behavior detection system based on videophone speech recognition, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the steps of the school bullying behavior detection method based on video phone speech recognition as described in any one of claims 1-7 are implemented when the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor.

Images