Campus bullying behavior detection method and system based on video phone voice recognition

Abstract

The invention relates to the technical field of voice recognition, in particular to a campus bullying behavior detection method and system based on video phone voice recognition. By acquiring the sound signals and the continuous image frames in the monitoring area of the video phone, effective fusion of visual and auditory information is realized, so that abnormal conditions in the campus are more comprehensively captured and analyzed. The feature points of the figures in the image frames are recognized, the position changes of the figures are tracked, the aggregation feature values of the figures in the monitoring area are obtained in combination with the position distribution of the figures, and the value is of great significance to the recognition of potential bullying scenes. When the aggregated feature value is abnormal, further analyzing the sound signal, dividing sound data segments, extracting acoustic features, and calculating an abnormal factor of each sound data segment; and finally, integrating the abnormal factors and the character aggregation characteristic values of all the sound data segments to obtain an abnormal characteristic value for monitoring the campus bullying behavior, thereby reducing the risk of missing report and false report.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition, and in particular to a method and system for detecting campus bullying behavior based on video phone speech recognition. Background Art

[0002] In the field of campus safety, the problem of campus bullying has always been the focus of attention of schools, parents and society. Campus bullying has caused serious harm to the physical and mental health of the victims, and also destroyed the harmonious atmosphere of the campus. By timely discovering and handling campus bullying, schools can better protect the physical and mental health of students, promote their all-round development, and provide students with a safer and more harmonious learning environment.

[0003] In recent years, with the rapid development of video surveillance technology, more and more schools have begun to use video surveillance to assist campus security management. However, simple video surveillance also faces some problems. Although video surveillance can capture images, it can only capture abnormal physical behaviors and makes insufficient use of sound information. Sound can often provide important clues. For example, abnormal sounds such as quarrels and crying may be related to campus bullying. Therefore, if the judgment of campus bullying behavior is based only on the surveillance images, verbal bullying and other behaviors may be ignored, resulting in reduced accuracy and credibility of campus bullying behavior detection. Summary of the invention

[0004] In order to solve the technical problem that judging campus bullying behavior only based on monitoring images will lead to insufficient utilization of sound information, thereby affecting the accuracy and credibility of campus bullying behavior detection, the purpose of the present invention is to provide a campus bullying behavior detection method and system based on video phone voice recognition, and the technical solution adopted is as follows:

[0005] Acquire the sound signals and multiple continuous image frames in the same period of time in the surveillance area of ​​the video phone;

[0006] Identify the feature points of the person in the image frame and track the position changes of the feature points in multiple consecutive image frames to quantify the movement feature value of the person; combine the movement feature value with the position distribution between the persons in all image frames to obtain the aggregation feature value between the persons;

[0007] The sound signal is divided according to the data fluctuation in the sound signal to obtain a plurality of sound data segments; based on the degree of deviation between the acoustic characteristics of each sound data segment and the standard situation, the abnormal factor of each sound data segment is determined;

[0008] The abnormal factors of all sound data segments and the aggregation characteristic values ​​between people in the monitoring area are combined to obtain abnormal characteristic values; based on the abnormal characteristic values, campus bullying behavior is detected to obtain detection results.

[0009] Furthermore, the method for obtaining the movement characteristic value includes:

[0010] In each of two adjacent image frames, the feature points of the person in the image frames are identified and matched to obtain a pair of matching feature points;

[0011] In each of two adjacent image frames, the Euclidean distance between the positions of two feature points in each pair of matching feature points is used as a displacement factor, and the average of the displacement factors between all matching feature point pairs in each of two adjacent image frames is used as a displacement parameter;

[0012] The average value of all displacement parameters corresponding to a plurality of consecutive image frames is taken as the movement feature value.

[0013] Furthermore, the method for obtaining the matching feature point pairs includes:

[0014] Each image frame is used as the input of a pre-trained neural network to obtain the person area in each image frame;

[0015] In each image frame, the SIFT algorithm is used to obtain the feature points in the human area and the feature values ​​corresponding to the feature points;

[0016] Combine each feature point in each image frame with each feature point in the next adjacent image frame to obtain all feature point pairs, and calculate the Euclidean distance between the feature values ​​of the two feature points in each feature point pair as a difference indicator;

[0017] For each feature point in each image, among all feature point pairs in the next adjacent image frame, the feature point pair corresponding to the minimum difference index is used as a matching feature point pair between the two adjacent image frames.

[0018] Furthermore, the method for obtaining the aggregated feature value includes:

[0019] In each image frame, all character regions are combined in pairs to obtain all region combinations;

[0020] In each area combination, the Euclidean distance between the centroids of two character areas is calculated as the distance factor, and the product of the mean of the distance factors corresponding to all area combinations and the minimum value of the distance factors in all area combinations is used as the distance parameter;

[0021] The ratio of the number of the human regions in each image frame to the distance parameter is used as a clustering factor between the human regions in each image frame;

[0022] The clustering feature value is determined by comprehensively considering the movement feature value and the clustering factors of all image frames, and the movement feature value is negatively correlated with the clustering feature value, and the clustering factor is positively correlated with the clustering feature value.

[0023] Furthermore, the method for obtaining the sound data segment includes:

[0024] The sound signal between every two adjacent extreme value points in the sound signal is taken as an initial data segment;

[0025] In each initial data segment, the numerical difference between the maximum value and the minimum value is taken as the fluctuation amplitude of each initial data segment;

[0026] Among all the initial data segments of the sound signal, the initial data segments are merged based on the similarity of the fluctuation amplitudes between adjacent initial data segments, so as to obtain all the sound data segments in the sound signal.

[0027] Furthermore, the merging process includes:

[0028] The absolute value of the difference between the fluctuation amplitudes of two adjacent initial data segments is negatively correlated and normalized to obtain the merged index;

[0029] In the sound signal, the first initial data segment in the time sequence is used as the data segment to be tested to start traversal, and it is determined whether the merging index between the data segment to be tested and the next adjacent initial data segment meets the preset condition. If not, the data segment to be tested is used as a sound data segment, and the next initial data segment adjacent to the data segment to be tested is used as a new data segment to be tested to continue traversal;

[0030] If the condition is satisfied, the next initial data segment adjacent to the data segment to be tested is merged into the data segment to be tested to obtain a new data segment to be tested, and it is determined whether the merging index between the last initial data segment in the new data segment to be tested and the next initial data segment adjacent thereto meets the preset condition. If the condition is not satisfied, the next initial data segment adjacent to the data segment to be tested is used as the new data segment to be tested to continue traversing;

[0031] Among them, the preset condition is that the merging index is greater than the preset merging threshold.

[0032] Furthermore, the method for obtaining the abnormal factor includes:

[0033] In each sound data segment, the difference between the mean of all maximum values ​​and the preset standard value is normalized to a value which is used as the sound amplitude deviation factor;

[0034] The ratio of the number of initial data segments in each sound data segment to the time length of the sound data segment is used as the fluctuation frequency in each sound data segment, and the difference between the fluctuation frequency and the preset standard frequency is normalized to a value as the fluctuation frequency deviation factor;

[0035] The product of the sound amplitude deviation factor and the fluctuation frequency deviation factor of each sound data segment is used as the abnormal factor of each sound data segment.

[0036] Furthermore, the method for obtaining the abnormal characteristic value includes:

[0037] The value obtained by normalizing the product of the mean value of the abnormal factors of all sound data segments and the aggregation characteristic value between the people in the monitoring area is used as the abnormal characteristic value.

[0038] Furthermore, the method for obtaining the detection result includes:

[0039] When the abnormal feature value is greater than or equal to the preset abnormal threshold, it is judged that bullying has occurred and an early warning is issued; when the abnormal feature value is less than the preset abnormal threshold, it is judged that bullying has not occurred.

[0040] A campus bullying behavior detection system based on video phone voice recognition includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and when the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor, the steps of the campus bullying behavior detection method based on video phone voice recognition are implemented.

[0041] The present invention has the following beneficial effects:

[0042] In view of the problem that the existing technology lacks the use of sound information when detecting campus bullying behavior, the present invention obtains the sound signal and multiple continuous image frames in the same period of the video phone monitoring area, so as to achieve the effective fusion of visual information and auditory information, and more comprehensively capture and analyze abnormal situations in the campus. In view of the fact that when campus bullying occurs, there is often physical contact or gathering of people, so by identifying the feature points of the characters in the image frame and tracking their position changes in continuous image frames, and combining the position distribution between the characters in the image frame, the aggregation feature value between the characters in the monitoring area is obtained, and the aggregation feature value is of great significance for identifying potential bullying scenes. It is not enough to accurately evaluate whether bullying behavior has occurred based solely on the aggregation feature value, so the sound signal is further analyzed. The sound signal is divided according to the data fluctuations in it to obtain sound data segments. At this time, each sound data segment can represent a continuous sound. At the same time, given that when bullying occurs, crying, quarreling, etc., are often accompanied by acoustic features such as high tone and fast speaking speed, the abnormal factor of each sound data segment is obtained based on the deviation between the acoustic features shown by each sound data segment and the standard situation, providing a strong basis for the judgment of bullying behavior. Finally, the abnormal factors of all sound data segments and the clustering feature values ​​between the characters in the monitoring area are combined to obtain the abnormal feature value. This comprehensive evaluation process can fully reflect the abnormal situation in the monitoring area, reduce the risk of missed reports and false alarms, and provide a more comprehensive and effective guarantee for campus safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A method flow chart of a campus bullying behavior detection method based on video phone voice recognition provided by an embodiment of the present invention;

[0045] Figure 2 A method flow chart of a method for obtaining a mobile characteristic value provided by an embodiment of the present invention;

[0046] Figure 3 A system block diagram of a campus bullying behavior detection system based on video phone voice recognition provided by an embodiment of the present invention;

[0047] Figure 4A schematic diagram of the system structure of a campus bullying behavior detection system based on video phone voice recognition provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a campus bullying behavior detection method and system based on video phone voice recognition proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0050] The following is a detailed description of a specific scheme of a campus bullying behavior detection method and system based on video phone voice recognition provided by the present invention in conjunction with the accompanying drawings.

[0051] See also Figure 1 , which shows a method flow chart of a campus bullying behavior detection method based on video phone voice recognition provided by an embodiment of the present invention, the method comprising the following steps:

[0052] Step S1: Acquire the sound signal and multiple consecutive image frames in the same period of time within the surveillance area of ​​the video phone.

[0053] In the crucial field of campus safety, the issue of campus bullying has long been the focus of common concern of school management, parents and all sectors of society. Campus bullying not only causes immeasurable harm to the physical and mental health of the victims, but also seriously undermines the harmony and tranquility that the campus should have. Therefore, timely discovery and proper handling of campus bullying is an important part of protecting students' physical and mental health, promoting their all-round development, and creating a safe and harmonious learning environment for schools.

[0054] In recent years, more and more schools have begun to use video surveillance technology to strengthen campus security management. However, video surveillance mainly focuses on abnormal physical behavior, while sound, an equally important source of information, is underutilized. In fact, sound can often provide key clues, such as quarrels, crying and other abnormal sounds, which may be closely related to school bullying.

[0055] Therefore, in an embodiment of the present invention, visual information is combined with auditory information. Specifically, video phone monitoring equipment with audio and video acquisition functions is installed in key areas of the campus (corridors, playgrounds, canteens, stairs) to ensure that the video phone can cover the key areas and the quality, resolution and audio acquisition capabilities of the video phone need to meet the needs of subsequent analysis. Then set the monitoring time period, and use the built-in audio acquisition module of the video phone to capture the sound signal in the monitoring area, and the sound signal is a time sequence signal with two dimensions of time and amplitude. At the same time, set a suitable frame rate, and use the camera part of the video phone to collect multiple continuous image frames in the same monitoring period in the monitoring area.

[0056] It should be noted that the duration of the monitoring time period can be set to 3 minutes, wherein the sampling frequency of the sound signal is set to 240HZ, and the image frame can be collected every 5s. The specific sampling frequency and the time interval for obtaining the image frame can be adjusted according to the implementation scenario and are not limited here. At the same time, the image frames and sound signals can also be denoised separately to enhance the image contrast and remove the interference of environmental noise in the sound signal. The sound signal can also be pre-emphasized to enhance the high-frequency components in the sound signal. The processes of denoising and pre-emphasis are both well-known technologies and will not be elaborated here.

[0057] Step S2: Identify the feature points of the person in the image frame and track the position changes of the feature points in multiple consecutive image frames to quantify the movement feature value of the person; combine the movement feature value with the position distribution between the persons in all image frames to obtain the aggregation feature value between the persons.

[0058] During school bullying, the distance between the perpetrator and the victim is relatively close, and the position will not move significantly within a period of time. Therefore, the image frames in the monitoring area can be analyzed to identify the feature points of the characters and track the position changes of the feature points in consecutive image frames, thereby reflecting the movement of the characters in the monitoring area and obtaining the movement feature value, which is an important indicator for evaluating potential bullying scenes. At the same time, since school bullying usually involves the gathering of people, there will also be characteristics of the gathering of people in the monitoring area. Therefore, the movement feature value is combined with the position distribution of the characters in all image frames to obtain the aggregation feature value between the characters in the monitoring area. The aggregation feature value analyzes the degree of aggregation of people in the monitoring area from two dimensions: the gathering of people and the movement of people, which helps to more accurately identify potential bullying scenes.

[0059] Tracking feature points is crucial for identifying people's behavior patterns. It can effectively distinguish individuals in complex backgrounds or crowded situations, thereby more accurately identifying people's movement trajectories, which helps to improve the accuracy of identifying school bullying scenes. Therefore, we first identify the feature points of the people in the image frame and track the position changes of the feature points in multiple consecutive image frames, and quantify the movement feature values ​​of the people in the image frames.

[0060] Preferably, in one embodiment of the present invention, the method for obtaining the mobile characteristic value includes:

[0061] See also Figure 2 , which shows a method flow chart of a method for obtaining a mobile characteristic value in one embodiment of the present invention, the method comprising the following steps:

[0062] Step S201: in every two adjacent image frames, identifying feature points of the person in the image frames and matching the feature points to obtain matching feature point pairs.

[0063] Each image frame is used as the input of a pre-trained neural network, and the neural network can automatically identify and mark the person area in the image, thereby obtaining the person area in each image frame.

[0064] Then, considering that the Scale-Invariant-Feature-Transform (SIFT) algorithm can detect local features in image frames, the SIFT algorithm is used to extract feature points in the character area identified in each image frame; at the same time, for each feature point, the local gradient of the image is measured in its surrounding neighborhood, and these gradient information are converted into a feature vector, that is, the feature value (descriptor) corresponding to each feature point is obtained. Each feature point in each image frame is combined with each feature point in the next adjacent image frame to obtain all feature point pairs. At this time, each feature point in each image frame corresponds to multiple feature point pairs (between the next adjacent image frame).

[0065] Next, the Euclidean distance between the eigenvalues ​​of the two feature points in each feature point pair is calculated as a difference index. The larger the Euclidean distance, the larger the difference index, which indicates that the difference between the two feature points in the feature space is greater. Conversely, the smaller the difference index, it means that the two feature points may be the representation of the same feature point in different image frames.

[0066] Finally, for the convenience of explanation and illustration, in each image, any feature point is selected as the target point. For the target point, among all the corresponding feature point pairs (between the adjacent next image frame), the feature point pair with the smallest difference index is taken as a matching feature point pair between the two adjacent image frames. At this time, the two feature points in the matching feature point pair in the two image frames are considered to represent the same feature.

[0067] It should be noted that the training process of the neural network is a process well known to those skilled in the art and will not be described in detail here; meanwhile, the SIFT algorithm is also a well-known technology and will not be described in detail here.

[0068] Step S202: In adjacent image frames, based on the positional relationship between two feature points in the matching feature point pair, a displacement parameter of the person moving in the two image frames is determined.

[0069] By matching the feature points in step S201, the position changes of the same person can be tracked across different image frames. Therefore, in each pair of adjacent image frames, the Euclidean distance between the positions of the two feature points in each pair of matching feature points is used as the displacement factor. The larger the displacement factor, the greater the change in the position of the feature point, and the more obvious the movement characteristics of the corresponding person. Finally, the mean of the displacement factors between all matching feature point pairs in the two adjacent image frames is used as the displacement parameter. At this time, the larger the displacement parameter, the greater the movement of the person in the two adjacent image frames, and the lower the possibility of representing a potential bullying scene.

[0070] It should be noted that when calculating the displacement factor, the same coordinate system can be constructed for all image frames. For example, the lower left corner of the image frame is taken as the origin, the horizontal right direction is the horizontal axis, and the vertical upward direction is the vertical axis. At this time, each feature point has a two-dimensional coordinate in the image frame to which it belongs, so as to calculate the displacement factor.

[0071] Step S203: synthesizing the displacement parameters of the character's movement between all image frames to obtain the movement feature value of the character in the image frame.

[0072] In the above steps, there is a displacement parameter between each pair of matching feature points in two adjacent image frames, which represents the movement of the characters in the two adjacent image frames. Therefore, the average of all displacement parameters corresponding to multiple consecutive image frames is taken as the movement feature value, denoted as YT. The movement feature value can represent the movement of all characters in the image frame during the monitoring period. The larger the movement feature value, the larger the movement amplitude of all characters in the image frame, which means that the characters are engaged in normal social activities, and the possibility of a bullying scene in the monitoring area is lower; conversely, the smaller the movement feature value, the smaller the movement amplitude of all characters in the image frame, suggesting some kind of static or concentrated behavior, and the possibility of a bullying scene in the monitoring area is higher.

[0073] The mobile feature value reflects the overall movement of people in the monitoring area, and can capture the activity patterns and dynamic changes of people, thus serving as an indicator for evaluating whether there is a bullying scene. In each image frame, the position distribution between people provides static spatial information, explaining the relative position and aggregation status of the people in space. Combining these two types of information can achieve a comprehensive consideration of the activities of people in the monitoring area and obtain the aggregation feature value, which facilitates a more accurate judgment of whether there is a potential bullying problem.

[0074] Preferably, in one embodiment of the present invention, the method for obtaining the aggregated feature value includes:

[0075] In each image frame, all character regions are combined in pairs to obtain all region combinations.

[0076] Then, in each area combination, the Euclidean distance between the centroids of the two character areas is calculated as the distance factor. The smaller the distance factor, the closer the distance between the two character areas. The product of the mean of the distance factors corresponding to all area combinations and the minimum value of the distance factors in all area combinations is taken as the distance parameter. The smaller the distance parameter, the smaller the distance between all characters in each image frame, that is, the higher the degree of aggregation, and the more likely it is to exceed the normal social distance, suggesting the existence of a bullying scene.

[0077] Since the bullying process not only involves the gathering of people, but also the gathering of crowds, and the degree of the gathering can be characterized by the number of character areas, the gathering factor between the characters in each image frame is calculated based on the distance parameter and the number of character areas. The more character areas there are, the more gathering of crowds has occurred. At the same time, if the distance parameter is smaller, it indicates that there is a gathering of people, and the possibility of a bullying scene is higher at this time. It should be noted that in this embodiment of the present invention, it is believed that in the same image frame, the centroids of different character areas will not overlap, that is, the value of the distance parameter is not 0. Therefore, in this embodiment of the present invention, the ratio of the number of character areas in each image frame to the distance parameter is used as the gathering factor between the characters in each image frame, denoted as JY. At this time, the larger the gathering factor, the higher the possibility of a bullying scene.

[0078] Finally, the moving eigenvalues ​​and the clustering factors of all image frames are combined to determine the clustering eigenvalues. The formula model of the clustering eigenvalues ​​includes:

[0079]

[0080] Where JT represents the aggregated feature value; n represents the number of image frames; JY i represents the clustering factor of the person in the i-th image frame; YT represents the moving feature value; γ1 represents the preset first parameter; norm() represents the normalization function.

[0081] In the formula model of the clustered eigenvalue, based on the above analysis, it can be seen that the larger the clustering factor, the higher the possibility of a bullying scene appearing in each image frame; at the same time, the smaller the mobile eigenvalue, the smaller the movement amplitude of all the characters in the image frame, suggesting some kind of static or concentrated behavior, then the possibility of a bullying scene existing in the monitoring area is higher, so the mean of the clustering factors of all image frames is taken as the numerator and the mobile eigenvalue is taken as the denominator to realize the above logical relationship, and the clustered eigenvalue is obtained after the ratio is normalized. At this time, the larger the clustered eigenvalue, the higher the possibility of bullying behavior occurring during the monitoring period.

[0082] It should be noted that the purpose of presetting the first parameter γ1 is to prevent the denominator from being 0, and the value may be 0.001. The specific value may be adjusted according to the implementation scenario and is not limited here.

[0083] At this point, through identifying and analyzing the images of the video phone, the possibility of school bullying was preliminarily determined.

[0084] Step S3: Divide the sound signal according to the data fluctuation in the sound signal to obtain multiple sound data segments; determine the abnormal factor of each sound data segment based on the degree of deviation between the acoustic characteristics of each sound data segment and the standard situation.

[0085] The clustering feature value indicates the gathering of people in the surveillance image. When school bullying occurs, the perpetrator will usually verbally humiliate the victim with a higher tone and faster speaking speed. The victim may also cry and the two sides may quarrel. These features can be used to analyze and quantify the sound signals, which helps to more accurately determine whether bullying has occurred during the monitoring period.

[0086] The sound signal is divided according to its data fluctuations to obtain sound data segments. At this time, each sound data segment can represent a continuous sound, which can make the sound features in each sound data segment more concentrated and consistent, and help improve the accuracy of subsequent processing.

[0087] Preferably, in one embodiment of the present invention, the method for obtaining the sound data segment includes:

[0088] The extreme value points in the sound signal are obtained and the sound signal is initially divided according to the extreme value points, that is, the sound signal between each two adjacent extreme value points is used as an initial data segment. At this time, each initial data segment obtained represents a fluctuation. It should be noted that the method for obtaining the extreme value points is a well-known technology and will not be described in detail here.

[0089] Then, in each initial data segment, the numerical difference between the maximum value and the minimum value is taken as the fluctuation amplitude of each initial data segment. The larger the fluctuation amplitude, the greater the fluctuation intensity of the sound.

[0090] Given that each initial data segment represents a fluctuation, and in real scenarios, continuous sounds with consistent features may be represented by multiple fluctuations in the sound signal, in all initial data segments of the sound signal, the similarity between the fluctuation amplitudes of each two adjacent initial data segments is analyzed to obtain a merging index, so that the initial data segments are merged according to the merging index to obtain all the sound data segments in the sound signal. At this time, each sound data segment can more accurately represent the characteristics of a continuous sound. Specifically, the absolute value of the difference between the two fluctuation amplitudes can be calculated. The larger the absolute value of the difference here, the greater the difference between the fluctuation amplitudes of the two adjacent initial data segments, the worse the consistency of the sound features, and the lower the possibility of merging. Therefore, the absolute value of the difference is negatively correlated and normalized as the merging index to achieve logical relationship correction. At this time, the larger the merging index, the more the two initial data segments should be merged. It should be noted that the negative correlation mapping and normalization method here can use the exp(-x) function, where exp represents an exponential function with the natural constant e as the base, and x represents an independent variable.

[0091] Finally, in the sound signal, the first initial data segment in the time sequence is used as the data segment to be tested to start traversal, and it is determined whether the merging index between the data segment to be tested and the next adjacent initial data segment meets the preset condition. If not, the data segment to be tested is used as a sound data segment, and the next initial data segment adjacent to the data segment to be tested is used as a new data segment to be tested to continue traversal;

[0092] If so, the next initial data segment adjacent to the data segment to be tested is merged into the data segment to be tested to obtain a new data segment to be tested, and a determination is made as to whether the merging index between the last initial data segment in the new data segment to be tested and the next adjacent initial data segment meets the preset condition; if not, the next initial data segment adjacent to the data segment to be tested is used as the new data segment to be tested to continue traversing.

[0093] After the traversal is completed, all the sound data segments can be obtained.

[0094] The preset condition is that the merging index is greater than the preset merging threshold. It should be noted that the preset merging threshold is set to 0.5, and the specific value can be adjusted according to the implementation scenario and is not limited here.

[0095] The above merging process is illustrated by an example: for example, there are 5 initial data segments in the sound signal, and these 5 initial data segments are arranged in time sequence as initial data segment 1, initial data segment 2, initial data segment 3, initial data segment 4, and initial data segment 5. At the beginning of the merging, the initial data segment 1 is used as the data segment to be tested, and it is determined whether the merging index between the data segment to be tested and the initial data segment 1 is greater than the merging threshold. If it is greater, the initial data segment 1 is merged with the initial data segment 2, and the merged data segment is used as the new data segment to be tested. At this time, the data index between the initial data segment 2 and the initial data segment 3 in the new data segment to be tested is continued to be determined. Whether the merging index of is greater than the merging threshold value, if not, the data segment after the initial data segment 1 and the initial data segment 2 are merged is taken as a sound data segment, and the initial data segment 3 is taken as a new data segment to be tested, and it is continued to be judged whether the merging index between the initial data segment 3 and the initial data segment 4 is greater than the merging threshold value, if not, the initial data segment 3 itself is directly taken as a sound data segment, and the initial data segment 4 is taken as a new data segment to be tested at this time and it is judged whether the merging index between the remaining initial data segments 5 is greater than the merging threshold value, if it is greater, the initial data segment 4 and the initial data segment 5 are merged as a new sound data segment.

[0096] When school bullying occurs, the perpetrator's voice usually becomes higher and his speech speed becomes faster. The victim may also cry and the two sides may quarrel. These acoustic characteristics can be manifested in the sound signal as large amplitude and high frequency of fluctuation. Therefore, by quantifying the degree of deviation between the acoustic characteristics of each sound data segment and the standard situation, the abnormal factor of each sound data segment can be obtained as another indicator to measure the possibility of school bullying during the monitoring period.

[0097] Preferably, in one embodiment of the present invention, the method for obtaining the abnormal factor includes:

[0098] Since in the campus bullying scene, the sound usually shows a high pitch and too fast speaking speed, both of which can be characterized by the amplitude and fluctuation frequency of the sound signal, so in the embodiment of the present invention, the acoustic features mainly analyze the amplitude and fluctuation frequency, and in each sound data segment, the difference between the mean of all maximum values ​​and the preset standard value is normalized to the value as the sound amplitude deviation factor, where the preset standard value can represent the intensity or amplitude of the sound signal in a normal scene, so if the difference between the mean of the maximum value and the preset standard value is larger, then the sound amplitude deviation factor is larger, indicating that the tone of the sound signal in the current monitoring area may be too high, and the possibility of bullying will also increase significantly. It should be noted that the difference here may be positive or negative, so the normalization method here uses the Sigmoid() function.

[0099] Since each initial data segment represents a data fluctuation in the sound signal, the ratio of the number of initial data segments in each sound data segment to the time length of the sound data segment is used as the fluctuation frequency in each sound data segment. The fluctuation frequency represents the number of times the sound wave vibrates. The larger the fluctuation frequency, the faster the speaker speaks. Then the difference between the fluctuation frequency and the preset standard frequency is normalized and used as the fluctuation frequency deviation factor. The preset standard frequency can represent the frequency of sound wave vibration in the sound signal under normal scenarios. Therefore, if the difference between the fluctuation frequency and the preset standard frequency is larger, the fluctuation frequency deviation factor is larger, which means that the speech speed of the sound signal in the current monitoring area may be too fast, resulting in frequent vibration of the sound signal, which also means that the possibility of bullying will increase significantly. It should be noted that the difference here may be positive or negative, so the normalization method here also uses the Sigmoid() function.

[0100] Based on the above analysis, it can be seen that the larger the sound amplitude deviation factor, the higher the possibility of bullying; the larger the fluctuation frequency deviation factor, the higher the possibility of bullying. Therefore, the sound amplitude deviation factor and the fluctuation frequency deviation factor are positively correlated with the possibility of bullying in the monitoring area. Therefore, in this embodiment of the present invention, the product of the sound amplitude deviation factor and the fluctuation frequency deviation factor of each sound data segment is used as the abnormal factor of each sound data segment. At this time, the larger the abnormal factor, the higher the possibility of bullying in the monitoring area under the auditory information.

[0101] It should be noted that the method for obtaining the preset standard value and the preset standard frequency can be: use a video phone to record a sound signal in the monitoring area under normal circumstances in advance, and then calculate the mean of all maximum values ​​in the sound signal as the preset standard value, and take the sound signal between adjacent extreme points as a data segment, and calculate the ratio of the number of data segments in the sound signal to the time length of the sound signal as the preset standard frequency. The specific values ​​of the preset standard value and the preset standard frequency can be adjusted according to the implementation scenario and are not limited here.

[0102] At this point, by identifying and analyzing the voice information of the video phone, the possibility of school bullying was further confirmed.

[0103] Step S4: Abnormal factors of all sound data segments and clustered feature values ​​between people in the monitoring area are integrated to obtain abnormal feature values; based on the abnormal feature values, campus bullying behavior is detected to obtain detection results.

[0104] In the above process, two indicators for judging whether school bullying occurs are analyzed and calculated from the visual and auditory perspectives respectively. In this step, these two indicators can be combined to obtain abnormal characteristic values, so as to more comprehensively measure the possibility of school bullying behavior occurring in the monitoring area and obtain the monitoring results of school bullying behavior.

[0105] Preferably, in one embodiment of the present invention, the method for obtaining abnormal feature values ​​includes:

[0106] Based on the analysis in step S2, it can be seen that the larger the aggregation eigenvalue, the more characters appear in the monitoring area under the visual information and the smaller the movement of all characters in the image frame, suggesting some kind of static or concentrated behavior, then the possibility of a bullying scene in the monitoring area is higher; at the same time, based on the analysis in step S3, it can be seen that the larger the abnormal factor, the higher the tone of the person's speech represented by the sound signal in the monitoring area under the auditory information, and the speaker's speaking speed is faster, then the possibility of bullying is higher.

[0107] Therefore, both the clustered feature value and the abnormal factor are positively correlated with the possibility of bullying in the monitoring area. Therefore, in this embodiment of the present invention, the product of the mean value of the abnormal factor of all sound data segments and the clustered feature value between the characters in the monitoring area is normalized and used as the abnormal feature value in the monitoring area. At this time, the larger the abnormal feature value, the higher the possibility of bullying in the monitoring area. Normalization is a technical means well known to those skilled in the art, and the normalization function can be linear normalization or standard normalization, etc. The specific normalization method is not limited here.

[0108] After obtaining the indicator to measure the possibility of bullying, judgment can be made based on the indicator, thereby detecting school bullying behavior and obtaining the test results.

[0109] Preferably, in one embodiment of the present invention, the method for obtaining the detection result includes:

[0110] When the abnormal characteristic value is greater than or equal to the preset abnormal threshold, it is judged that bullying has occurred and an early warning is issued. At the same time, a call can be made to the contact person or administrator to facilitate the contact person or administrator to deal with campus bullying in a timely manner. When the abnormal characteristic value is less than the preset abnormal threshold, it is judged that bullying has not occurred and no early warning is required.

[0111] It should be noted that the preset abnormal threshold is 0.65, and the specific value can be adjusted according to the implementation scenario and is not limited here.

[0112] In summary, in view of the problem that the prior art lacks the use of sound information when detecting campus bullying, the embodiment of the present invention obtains the sound signal and multiple continuous image frames in the same period of the video phone monitoring area, so as to achieve the effective fusion of visual information and auditory information, and more comprehensively capture and analyze abnormal situations in the campus. In view of the fact that when campus bullying occurs, there is often physical contact or gathering of people, so by identifying the feature points of the characters in the image frame and tracking their position changes in the continuous image frames, and combining the position distribution between the characters in the image frame, the aggregation feature value between the characters in the monitoring area is obtained, and the aggregation feature value is of great significance for identifying potential bullying scenes. It is not enough to accurately evaluate whether bullying behavior has occurred based solely on the aggregation feature value, so the sound signal is further analyzed. The sound signal is divided according to the data fluctuations in it to obtain sound data segments. At this time, each sound data segment can represent a continuous sound. At the same time, given that when bullying occurs, crying, quarreling, etc., are often accompanied by acoustic features such as high tone and fast speaking speed, the abnormal factor of each sound data segment is obtained based on the deviation between the acoustic features shown by each sound data segment and the standard situation, providing a strong basis for the judgment of bullying behavior. Finally, the abnormal factors of all sound data segments and the clustering feature values ​​between the characters in the monitoring area are combined to obtain the abnormal feature value. This comprehensive evaluation process can fully reflect the abnormal situation in the monitoring area, reduce the risk of missed reports and false alarms, and provide a more comprehensive and effective guarantee for campus safety.

[0113] The embodiment of the present invention also proposes a campus bullying behavior detection system based on video phone voice recognition, please refer to Figure 3 , which shows a system block diagram of a campus bullying behavior detection system based on video phone voice recognition, including: a data acquisition module 301, used to implement step S1 in the above method; a character feature analysis module 302, used to implement step S2 in the above method; a sound abnormal feature analysis module 303, used to implement step S3 in the above method; a campus bullying behavior detection module 304, used to implement step S4 in the above method.

[0114] It should be noted that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the campus bullying behavior detection system based on video phone voice recognition and the campus bullying behavior detection method based on video phone voice recognition provided in the above embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0115] See also Figure 4 , which shows a system structure diagram of a campus bullying behavior detection system based on video phone voice recognition provided by an embodiment of the present invention, including a processor 400, a memory 401, a bus 402 and a communication interface 403, wherein the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; wherein the memory 401 may include a high-speed random access memory, the bus 402 may be an ISA bus, a PCI bus or an EISA bus, etc., and the processor 400 may be an integrated circuit chip with signal processing capabilities; the memory 401 stores at least one instruction, at least one program, a code set or an instruction set, and when the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor, the steps in a campus bullying behavior detection method based on video phone voice recognition are implemented.

[0116] It should be noted that the sequence of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0117] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for detecting campus bullying behavior based on video phone voice recognition, characterized in that: The method comprises: Acquire the sound signals and multiple continuous image frames in the same period of time in the surveillance area of ​​the video phone; Identify the feature points of the person in the image frame and track the position changes of the feature points in multiple consecutive image frames to quantify the movement feature value of the person; combine the movement feature value with the position distribution between the persons in all image frames to obtain the aggregation feature value between the persons; The sound signal is divided according to the data fluctuation in the sound signal to obtain a plurality of sound data segments; based on the degree of deviation between the acoustic characteristics of each sound data segment and the standard situation, the abnormal factor of each sound data segment is determined; The abnormal factors of all sound data segments and the aggregation characteristic values ​​between people in the monitoring area are combined to obtain abnormal characteristic values; based on the abnormal characteristic values, campus bullying behavior is detected to obtain detection results.

2. According to claim 1, a method for detecting campus bullying behavior based on video phone voice recognition is characterized in that: The method for obtaining the mobile characteristic value comprises: In each of two adjacent image frames, the feature points of the person in the image frames are identified and matched to obtain a pair of matching feature points; In each of two adjacent image frames, the Euclidean distance between the positions of two feature points in each pair of matching feature points is used as a displacement factor, and the average of the displacement factors between all matching feature point pairs in each of two adjacent image frames is used as a displacement parameter; The average value of all displacement parameters corresponding to a plurality of consecutive image frames is taken as the movement feature value.

3. The campus bullying behavior detection method based on video phone voice recognition according to claim 2 is characterized in that: The method for obtaining the matching feature point pair includes: Each image frame is used as the input of a pre-trained neural network to obtain the person area in each image frame; In each image frame, the SIFT algorithm is used to obtain the feature points in the human area and the feature values ​​corresponding to the feature points; Combine each feature point in each image frame with each feature point in the next adjacent image frame to obtain all feature point pairs, and calculate the Euclidean distance between the feature values ​​of the two feature points in each feature point pair as a difference indicator; For each feature point in each image, among all feature point pairs in the next adjacent image frame, the feature point pair corresponding to the minimum difference index is used as a matching feature point pair between the two adjacent image frames.

4. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 3, characterized in that: The method for obtaining the aggregated feature value comprises: In each image frame, all character regions are combined in pairs to obtain all region combinations; In each area combination, the Euclidean distance between the centroids of two character areas is calculated as the distance factor, and the product of the mean of the distance factors corresponding to all area combinations and the minimum value of the distance factors in all area combinations is used as the distance parameter; The ratio of the number of the human regions in each image frame to the distance parameter is used as a clustering factor between the human regions in each image frame; The clustering feature value is determined by comprehensively considering the movement feature value and the clustering factors of all image frames, and the movement feature value is negatively correlated with the clustering feature value, and the clustering factor is positively correlated with the clustering feature value.

5. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 1, characterized in that: The method for obtaining the sound data segment includes: The sound signal between every two adjacent extreme value points in the sound signal is taken as an initial data segment; In each initial data segment, the numerical difference between the maximum value and the minimum value is taken as the fluctuation amplitude of each initial data segment; Among all the initial data segments of the sound signal, the initial data segments are merged based on the similarity of the fluctuation amplitudes between adjacent initial data segments, so as to obtain all the sound data segments in the sound signal.

6. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 5, characterized in that: The merging process includes: The absolute value of the difference between the fluctuation amplitudes of two adjacent initial data segments is negatively correlated and normalized to obtain the merged index; In the sound signal, the first initial data segment in the time sequence is used as the data segment to be tested to start traversal, and it is determined whether the merging index between the data segment to be tested and the next adjacent initial data segment meets the preset condition. If not, the data segment to be tested is used as a sound data segment, and the next initial data segment adjacent to the data segment to be tested is used as a new data segment to be tested to continue traversal; If the condition is satisfied, the next initial data segment adjacent to the data segment to be tested is merged into the data segment to be tested to obtain a new data segment to be tested, and it is determined whether the merging index between the last initial data segment in the new data segment to be tested and the next initial data segment adjacent thereto meets the preset condition. If the condition is not satisfied, the next initial data segment adjacent to the data segment to be tested is used as the new data segment to be tested to continue traversing; Among them, the preset condition is that the merging index is greater than the preset merging threshold.

7. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 5, characterized in that: The method for obtaining the abnormal factor includes: In each sound data segment, the difference between the mean of all maximum values ​​and the preset standard value is normalized to a value which is used as the sound amplitude deviation factor; The ratio of the number of initial data segments in each sound data segment to the time length of the sound data segment is used as the fluctuation frequency in each sound data segment, and the difference between the fluctuation frequency and the preset standard frequency is normalized to a value as the fluctuation frequency deviation factor; The product of the sound amplitude deviation factor and the fluctuation frequency deviation factor of each sound data segment is used as the abnormal factor of each sound data segment.

8. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 1, characterized in that: The method for obtaining the abnormal characteristic value comprises: The value obtained by normalizing the product of the mean value of the abnormal factors of all sound data segments and the aggregation characteristic value between the people in the monitoring area is used as the abnormal characteristic value.

9. The method for detecting campus bullying behavior based on video phone voice recognition according to claim 1, characterized in that: The method for obtaining the detection result includes: When the abnormal feature value is greater than or equal to the preset abnormal threshold, it is judged that bullying has occurred and an early warning is issued; when the abnormal feature value is less than the preset abnormal threshold, it is judged that bullying has not occurred.

10. A campus bullying behavior detection system based on video phone voice recognition, characterized in that: It includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and when the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor, the steps of a campus bullying behavior detection method based on video phone voice recognition as described in any one of claims 1 to 9 are implemented.

Images