An abnormal behavior recognition method and system based on sound and image fusion information

Abstract

The application relates to an abnormal behavior recognition method and system based on sound and image fusion information, and relates to the field of behavior recognition, which comprises the following steps: collecting a scene image; identifying the number of personnel from the scene image; identifying the personnel distribution from the scene image when the number of personnel is greater than 1; determining a moving vector according to the personnel distribution; determining a direction dispersion in response to the moving vector; collecting a scene sound if the direction dispersion is greater than a preset disorder threshold value; identifying a high-pitched sound source from the scene sound; calculating the quotient of the high-pitched sound source and the number of personnel, and defining the quotient as a high-pitched proportion; and generating and displaying an abnormal behavior prompt based on the high-pitched proportion when the high-pitched proportion is greater than a preset noisy threshold value. The application has the effects of improving the accuracy of abnormal behavior recognition and accurately recognizing the behavior of a target in a multi-target scene.

Description

Technical Field

[0001] This invention relates to the field of behavior recognition, and in particular to a method and system for identifying abnormal behavior based on the fusion of sound and image information. Background Technology

[0002] Abnormal behavior recognition refers to a technical system that collects target scene data through sensors (such as cameras, radar, and sensor networks), analyzes the target's behavior patterns by combining them with algorithm models, identifies behaviors that "deviate from normal / expected behavioral trajectories, rules, or characteristics," and triggers early warnings or linkage control.

[0003] In existing technologies, image recognition technology is generally used to identify the behavior of a target. First, a clear and complete image of the target is acquired through a sensor, and then the target's behavior is identified from the image to filter out abnormal behaviors. Once abnormal behaviors are detected, staff are notified in a timely manner.

[0004] When the scene in which the target is located is crowded, the target is easily obscured, resulting in an incomplete image of the target, which makes it difficult for image recognition technology to identify the target's behavior. Summary of the Invention

[0005] To improve the accuracy of abnormal behavior recognition and to accurately identify the behavior of targets in multi-target scenarios, this invention provides an abnormal behavior recognition method and system based on the fusion of sound and image information.

[0006] In a first aspect, the present invention provides a method for identifying abnormal behavior based on the fusion of sound and image information, employing the following technical solution: An abnormal behavior recognition method based on the fusion of sound and image information includes: Step 100: Acquire scene images; Step 101: Identify the number of people from the scene image; Step 102: When the number of people is greater than 1, identify the distribution of people from the scene image; Step 103: Determine the movement vector based on the personnel distribution; Step 104: Determine the orientation dispersion in response to the movement vector; Step 105: If the directional dispersion is greater than a preset disorder threshold, collect scene sound; Step 106: Identify the high-pitched sound source from the scene sounds; Step 107: Calculate the quotient of the high-pitched sound source and the number of people, and define it as the high-pitched percentage; Step 108: When the proportion of high-pitched sounds is greater than a preset noise threshold, generate and display an abnormal behavior prompt based on the proportion of high-pitched sounds.

[0007] By adopting the above technical solution, the number of people in the target scene is detected in real time. When there are many people, the changes in their positions over time are compared to determine their movement. Then, the directional dispersion is calculated based on the movement of people to determine the consistency of their movement. When the consistency of people's movement is poor, it is determined that the people in the scene are relatively chaotic, and abnormal behavior prompts are issued in a timely manner to notify the staff. In this way, the detection of group behavior is replaced by the detection of individual behavior, thereby improving the accuracy of abnormal behavior identification.

[0008] When the consistency of personnel movement is poor, the sound situation in the target scene is detected to identify the number of loud sound sources, and the number of people is compared with the number of people to determine the degree of chaos in the target scene, thereby improving the accuracy of abnormal behavior identification.

[0009] Optional, also includes: Step 109: If the directional dispersion is not greater than a preset disorder threshold, determine the main direction in response to the movement vector, and determine the scene position based on the scene image; Step 110: Determine the main address based on the scene location and main direction; Step 111: Determine the carrying capacity in response to the primary address; Step 112: Determine the baseline flow rate based on the carrying capacity, and determine the moving flow rate based on the moving vector; Step 113: When the moving flow rate is greater than the reference flow rate, determine the flow retardation coefficient by combining the moving flow rate and the reference flow rate; Step 114: In response to the generation of the flow slowing coefficient, display a flow slowing prompt for personnel.

[0010] By adopting the above technical solution, when the consistency of personnel movement is good, the main address of the personnel can be identified, and the personnel carrying capacity of the main address can be retrieved, thereby judging the congestion situation of the main address, and issuing timely prompts when the main address is relatively congested to control the flow of people entering the main address.

[0011] Optionally, a fall detection method may also be included, the fall detection method comprising: Step 200: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the personnel density based on the personnel distribution; Step 201: Determine the isolated location in response to the population density; Step 202: Determine the isolated vector based on the isolated location; Step 203: Determine the degree of deviation by combining the isolated vector and the main direction; Step 204: If the deviation is greater than a preset deviation threshold, identify the person's posture from the scene image based on the isolated position; Step 205: Determine the fall location in response to the person's posture and a preset fall posture; Step 206: Generate and display a fall notification based on the fall location.

[0012] By adopting the above technical solution, when people are in a chaotic situation, they are prone to collisions and falls. The solution identifies isolated individuals in the target scene who are surrounded by few people and determines the deviation of their direction of movement and main address. When the deviation is large, the solution can detect the isolated individual's posture in real time and identify when they fall, thereby improving the accuracy of abnormal behavior identification.

[0013] Optionally, the fall detection method further includes: Step 207: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the crowded area based on the population density; Step 208: Determine the congestion direction based on the congested area; Step 209: Determine congestion deviation in response to the congestion direction; Step 210: Determine the reverse position based on the congestion deviation; Step 211: Determine the attenuation coefficient based on the reverse position; Step 212: Determine the attenuation characteristics by combining the attenuation coefficient and the preset fall characteristics; Step 213: If there is attenuation in the scene sound, define the reverse position as the fall position.

[0014] By adopting the above technical solution, when people are in a chaotic situation, the crowded areas where people gather can be identified, and the direction of movement of people in the crowded areas can be identified. By comparing the direction of crowding with the direction of movement of people, people moving against the flow of traffic in the crowded areas can be identified. In addition, sound information can be used to determine whether there have been falls, thereby improving the accuracy of abnormal behavior identification.

[0015] Optionally, the fall detection method further includes: Step 214: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the number of people in the crowd and the edge of the crowd based on the crowded area; Step 215: Determine the number of people entering and exiting in response to the congestion edge; Step 216: Calculate the sum of the preset periodic number of people and the number of people entering and exiting, and define it as the real-time number of people; Step 217: If the real-time number of people is greater than the number of people in the crowd, determine the trajectory of people based on the crowded area; Step 218: Identify the avoidance location from the personnel trajectory; Step 219: Define the avoidance position as the fall position.

[0016] By adopting the above technical solution, when people gather, the number of people entering and leaving the crowded area can be identified from the image, thereby calculating the real-time number of people in the crowded area. The actual number of people in the crowd can be identified from the image, and when the real-time number of people is greater than the number of people in the crowd, it is determined that someone has fallen and is being blocked by other people. At this time, the trajectory of other people in the crowded area is compared to identify the avoidance position where there is an avoidance situation, thereby determining that there is a person who has fallen at the avoidance position, thus improving the accuracy of abnormal behavior identification.

[0017] Optionally, it also includes a fall handling method, the fall handling method comprising: Step 300: Identify the work distribution from the scene image; Step 301: Select an assistance location from the work distribution based on the fall location; Step 302: Generate assistance information in response to the assistance location and the fall location, and determine the assistance number based on the assistance location; Step 303: Send the assistance information according to the assistance number to notify staff to assist the person who has fallen.

[0018] By adopting the above technical solution, when a person falls, the system identifies the staff in the target scene, selects the assistance position of the staff member closest to the fall location, plans the route from the assistance position to the fall location, and sends it to the staff member at the assistance position, thereby improving the efficiency of assisting the fallen person.

[0019] Optionally, the fall handling method further includes: Step 304: Determine the assistance distance based on the assistance location and the fall location; Step 305: When the assistance distance is greater than a preset rapid threshold, the approach flow rate is determined by combining the movement vector and the fall location, and the fall capacity is determined based on the fall location; Step 306: Calculate the quotient of the proximity flow and the fall capacity, and define it as the congestion coefficient; Step 307: If the congestion coefficient is greater than the preset interference threshold, determine the proximity address based on the proximity flow. Step 308: Determine a detour route in response to the fall location and proximity address; Step 309: Generate and display address detour information based on the detour route.

[0020] By adopting the above technical solution, when the aid location is far from the fall location, the approach flow near the fall location is identified. When the approach flow is large, it is determined that there is a risk of stampede. At this time, the approach address of people near the fall location is identified according to the movement of people, and a detour route is planned to bypass the fall location to reach the approach address, thereby reducing the situation where too many people at the fall location cause secondary injuries to the fall victim.

[0021] Optionally, the fall handling method further includes: Step 310: If the congestion coefficient is greater than the preset interference threshold, retrieve the historical trajectory based on the proximity flow. Step 311: Identify the source proportion from the historical trajectory; Step 312: Determine the primary source in response to the stated source percentage; Step 313: Determine the access equipment based on the scene location and main source; Step 314: Generate a passage control signal based on the passage equipment and fall detection capacity; Step 315: Control the passage device according to the passage control signal.

[0022] By adopting the above technical solution, when there is a large flow of people, the main source of people approaching the fall location can be identified according to the movement of people. Then, passage equipment that controls the passage between the main source and the fall location can be matched. The flow of people can be controlled by the passage equipment according to the carrying capacity of the fall location to reduce the situation where too many people at the fall location cause secondary injuries to the fall location.

[0023] Secondly, this application provides an abnormal behavior recognition system based on the fusion of sound and image information, employing the following technical solution: An abnormal behavior recognition system based on the fusion of sound and image information includes: The acquisition module is used to acquire scene images and scene sounds; The memory is used to store the program for any of the above-mentioned abnormal behavior recognition methods based on the fusion of sound and image information; The processor is the unit of memory that allows programs to be loaded and executed by the processor.

[0024] By adopting the above technical solution, the number of people in the target scene is detected in real time. When there are many people, the changes in their positions over time are compared to determine their movement. Then, the directional dispersion is calculated based on the movement of people to determine the consistency of their movement. When the consistency of people's movement is poor, it is determined that the people in the scene are relatively chaotic, and abnormal behavior prompts are issued in a timely manner to notify the staff. In this way, the detection of group behavior is replaced by the detection of individual behavior, thereby improving the accuracy of abnormal behavior identification.

[0025] In summary, this application includes at least one of the following beneficial technical effects: 1. Real-time detection of the number of people in the target scene, and comparison of the changes in the positions of people over time when there are many people to determine the movement of people, and then calculation of the directional dispersion based on the movement of people to determine the consistency of people's movement, and when the consistency of people's movement is poor, it is determined that the people in the scene are relatively chaotic, and abnormal behavior prompts are issued in a timely manner to notify the staff, thereby replacing the detection of individual behavior with the behavior of the group, thus improving the accuracy of abnormal behavior identification. 2. When the consistency of personnel movement is poor, the sound situation in the target scene is detected to identify the number of loud sound sources and compare them with the number of personnel to determine the degree of chaos in the target scene, thereby improving the accuracy of abnormal behavior identification. 3. When the consistency of personnel movement is relatively good, identify the main addresses of the personnel and retrieve the personnel carrying capacity of the main addresses to determine the congestion situation of the main addresses. When the main addresses are relatively congested, issue timely prompts to control the flow of people entering the main addresses. Attached Figure Description

[0026] Figure 1 This is a flowchart of an abnormal behavior recognition method based on the fusion of sound and image information; Figure 2 This is a flowchart of the fall detection method; Figure 3 This is a flowchart of the fall handling procedure. Detailed Implementation

[0027] To make the address, technical solution, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0028] Reference Figure 1 An abnormal behavior recognition method based on the fusion of sound and image information includes: Step 100: Acquire scene images.

[0029] Scene images refer to images of the target scene, such as shopping malls, schools, and factories, where abnormal behavior recognition is required. Scene images can be captured by cameras fixed to the target scene. The method of capturing scene images is selected by the staff according to the actual situation, and will not be elaborated here.

[0030] Step 101: Identify the number of people from the scene image.

[0031] The number of people refers to the number of people in a scene. The number of people can be identified through image recognition technology. The method for identifying the number of people is common knowledge in this field and will not be elaborated here.

[0032] Step 102: When the number of people is greater than 1, identify the distribution of people from the scene image.

[0033] A number of people greater than 1 indicates the presence of multiple targets in the scene. In this case, people are prone to occlusion, making it difficult to capture a complete image of each person. Person distribution refers to the position of each person in the scene. The position of each person can be identified by image recognition technology as the person distribution. The method for identifying the person distribution is common knowledge in the field and will not be elaborated here.

[0034] Step 103: Determine the movement vector based on the personnel distribution.

[0035] A movement vector is a vector used to show the movement of people in a target scene. The movement vector can be determined by comparing the changes in the positions of people in the distribution of people. That is, the vector pointing from the position of people in the previous period to the position of people in the current period is used as the movement vector. The method of determining the movement vector is common knowledge in the field and will not be elaborated here.

[0036] Step 104: Determine the directional dispersion in response to the movement vector.

[0037] Directional dispersion refers to a numerical value used to demonstrate the consistency of personnel movement. The more disordered the movement vector, the greater the directional dispersion. First, the movement vector can be converted into movement coordinates on a unit circle (i.e., calculate the cosine and sine components). Then, the average value of the movement coordinates is calculated as the average vector. Next, the average magnitude of the average vector is calculated (the value range of the average magnitude is [0, 1]. An average magnitude of 1 indicates that all personnel are in the same direction, and an average magnitude of 0 indicates that the directions are evenly distributed and completely disordered). Then, the circumferential standard deviation of the average magnitude is calculated (CSD = √(-2 × ln(R))). Finally, the circumferential standard deviation is normalized to obtain the directional dispersion. The calculation method of directional dispersion is common knowledge in this field and will not be elaborated here.

[0038] Step 105: If the directional dispersion is greater than the preset disorder threshold, collect scene sound.

[0039] The disorder threshold refers to the minimum directional dispersion of personnel movement with low consistency. The disorder threshold is selected by staff based on the actual situation and will not be elaborated upon here. A directional dispersion greater than the disorder threshold indicates relatively disordered personnel movement, which can easily lead to collisions and other accidents. Scene sound refers to the sound information in the target scene. Scene sound can be collected using microphones evenly distributed within the target scene. The method for collecting scene sound is selected by staff based on the actual situation and will not be elaborated upon here.

[0040] The system detects the number of people in a target scene in real time. When there are many people, it compares the changes in their positions over time to determine their movement. Then, it calculates the directional dispersion based on the movement to determine the consistency of the movement. When the consistency of the movement is poor, it indicates that the scene is chaotic and issues abnormal behavior alerts to notify staff in a timely manner. This improves the accuracy of abnormal behavior identification by replacing the detection of individual behavior with the detection of group behavior.

[0041] Step 106: Identify the high-pitched sound source from the scene sound.

[0042] High-volume sound sources refer to the number of locations with relatively high volume sound sources in a scene. The volume of people emitting sound can be identified from the scene's sound, and the location of the person emitting the sound can be determined by combining the positions of different microphones. The volume can then be adjusted according to the person's position to identify sound information with different volumes emanating from different locations. Locations with volumes exceeding a preset high volume threshold are selected as sound source locations. Finally, the number of sound source locations is counted as high-volume sound sources. The high volume threshold refers to the sound source with relatively high volume. The high volume threshold is selected by the staff based on the actual situation. The method for identifying high-volume sound sources is common knowledge in the field and will not be elaborated here.

[0043] Step 107: Calculate the quotient of the high-pitched sound source and the number of people, and define it as the high-pitched percentage.

[0044] The treble ratio is a numerical value used to show the level of noise of people in a target scene. The higher the treble ratio, the noisier the people are. The calculation method of the treble ratio is common knowledge in the field and will not be elaborated here.

[0045] Step 108: When the proportion of high-pitched sounds is greater than a preset noise threshold, generate and display an abnormal behavior prompt based on the proportion of high-pitched sounds.

[0046] The noise threshold refers to the minimum percentage of high-pitched sounds that indicate a relatively noisy environment. This threshold is selected by staff based on the actual situation and will not be elaborated upon here. A high-pitched sound percentage greater than the noise threshold indicates a relatively noisy or chaotic target environment. Abnormal behavior alerts will display information about the consistency of personnel to staff. The method for generating abnormal behavior alerts is common knowledge in this field and will not be elaborated upon here.

[0047] When the consistency of personnel movement is poor, the sound situation in the target scene is detected to identify the number of loud sound sources, and the number of people is compared with the number of people to determine the degree of chaos in the target scene, thereby improving the accuracy of abnormal behavior identification.

[0048] An abnormal behavior recognition method based on the fusion of sound and image information further includes: Step 109: If the directional dispersion is not greater than a preset disorder threshold, determine the main direction in response to the movement vector, and determine the scene position based on the scene image.

[0049] The main direction refers to the direction that people in the target scene mainly go, which is the angle value corresponding to the average vector mentioned above. The calculation method of the main direction is the same as step 104 above, and will not be repeated here.

[0050] Scene location refers to the location of the target scene. The scene location can be determined by image recognition technology. The method of scene location recognition is common knowledge to those in the field and will not be elaborated here.

[0051] Step 110: Determine the main address by combining the scene location and main direction.

[0052] The primary address refers to the space that people in the target scene mainly visit. The spatial distribution around the target scene can be retrieved to determine the primary address. The spatial distribution can be pre-entered by the staff. The method for determining the primary address is selected by the staff according to the actual situation, which will not be elaborated here.

[0053] Step 111: Determine the carrying capacity in response to the primary address.

[0054] The carrying capacity refers to the maximum number of people that a primary address can support. The carrying capacity corresponding to a primary address can be found in the capacity mapping table, which is a data table that records different primary addresses and their corresponding carrying capacities.

[0055] Step 112: Determine the baseline flow rate based on the carrying capacity, and determine the moving flow rate based on the moving vector.

[0056] The baseline capacity refers to the maximum number of people that can enter a main address per unit of time. The baseline capacity corresponding to the capacity can be found in the capacity mapping table. The capacity mapping table is a data table that records different capacity and their corresponding baseline capacity.

[0057] Mobile traffic refers to the number of people in a target scenario who enter the main address within a unit of time. Mobile traffic can be determined by superimposing movement vectors based on the number of cycles within a unit of time. For example, when the update cycle for the distribution of people is 10 seconds and the unit of time is 1 minute, 6 corresponding movement vectors are superimposed at each person's location to predict the movement of people, and the number of people arriving at the main target is taken as the mobile traffic. The method for determining mobile traffic is common knowledge in the field and will not be elaborated here.

[0058] Step 113: When the moving flow is greater than the reference flow, determine the flow retardation coefficient by combining the moving flow and the reference flow.

[0059] Mobile traffic exceeding the baseline traffic indicates that too many people are entering the main address per unit time, which can easily lead to congestion. The flow control factor is a value used to regulate the number of people entering the main address. The larger the flow control factor, the more the number of people entering the main address needs to be reduced. The flow control factor can be calculated as the quotient of mobile traffic and baseline traffic. The calculation method for the flow control factor is selected by the staff according to the actual situation, and will not be elaborated here.

[0060] Step 114: In response to the generation of the flow slowing coefficient, display a flow slowing prompt for personnel.

[0061] Crowd control prompts are information used to show staff the crowding situation. The method for generating crowd control prompts is common knowledge in the field and will not be elaborated here.

[0062] When the consistency of personnel movement is good, the main addresses of the personnel are identified, and the personnel carrying capacity of the main addresses is retrieved to determine the congestion situation of the main addresses. When the main addresses are relatively congested, timely prompts are issued to control the flow of people entering the main addresses.

[0063] Reference Figure 2 Fall detection methods include: Step 200: When the proportion of high-pitched sounds is greater than the preset noise threshold, determine the personnel density based on the personnel distribution.

[0064] Crowd density refers to a numerical value used to show the density of people in a target scene, that is, the number of people in a certain area around each person. Each person's location has a corresponding crowd density. The calculation method of crowd density is common knowledge in this field and will not be elaborated here.

[0065] Step 201: Determine the isolated location in response to the population density.

[0066] An isolated location refers to a location where there are relatively few people around, that is, a location with a population density below the isolation threshold. The isolation threshold is the maximum population density where there are relatively few people. The isolation threshold is selected by the staff based on the actual situation, and will not be elaborated here.

[0067] Step 202: Determine the isolated vector based on the isolated location.

[0068] An isolated vector is a movement vector corresponding to an isolated position. The method for retrieving an isolated vector is selected by the staff according to the actual situation, and will not be elaborated here.

[0069] Step 203: Determine the degree of deviation by combining the isolated vector and the main direction.

[0070] The degree of deviation refers to the numerical value used to show the deviation of the movement of a person in an isolated position from the main direction. The isolated direction can be extracted from the isolated vector first, and then the difference between the isolated direction and the main direction can be calculated as the degree of deviation. The calculation method of the degree of deviation is common knowledge in the field and will not be elaborated here.

[0071] Step 204: If the deviation is greater than a preset deviation threshold, identify the person's posture from the scene image based on the isolated position.

[0072] The deviation threshold refers to the minimum degree of deviation that a person in an isolated position is considered to have seriously deviated from the main direction. A deviation of 60 degrees is generally used as the deviation threshold. The deviation threshold is selected by the staff based on the actual situation and will not be elaborated here. A deviation greater than the deviation threshold means that the person in the isolated position is likely to collide with other people when moving, leading to a fall. Person posture refers to the posture data of a person in an isolated position. Person posture can be determined through posture recognition technology. The method for recognizing person posture is common knowledge in the field and will not be elaborated here.

[0073] Step 205: Determine the fall location in response to the person's posture and the preset fall posture.

[0074] Fall posture refers to the body posture data of a person when falling, such as the characteristic points of the arm supporting the ground (palm, elbow) and the posture when they touch the ground. The fall posture is selected by the staff according to the actual situation, and will not be elaborated here.

[0075] The fall location is the isolated position corresponding to the body posture of the person who fell. The method for determining the fall location is common knowledge in this field and will not be elaborated here.

[0076] Step 206: Generate and display a fall notification based on the fall location.

[0077] A fall alert is a message used to notify staff of a fall when someone in an isolated position has fallen. The method for generating a fall alert is common knowledge in the field and will not be elaborated here.

[0078] When people are in a chaotic situation, they are prone to colliding with each other and falling. The system identifies isolated individuals in a target scene with few people around them and determines the degree of deviation of these individuals from their main location and direction of movement. When the deviation is significant, the system can detect the posture of isolated individuals in real time and identify falls, thereby improving the accuracy of abnormal behavior identification.

[0079] Fall detection methods also include: Step 207: When the proportion of high-pitched sounds is greater than the preset noise threshold, determine the crowded area based on the population density.

[0080] A congested area refers to a region where people are relatively crowded, that is, a region where the population density is greater than the congestion threshold. The congestion threshold is the minimum population density at which people are relatively crowded. The congestion threshold is selected by the staff based on the actual situation. The method for determining the congested area is common knowledge in this field and will not be elaborated here.

[0081] Step 208: Determine the congestion direction based on the congestion area.

[0082] The direction of congestion refers to the main direction of people in the congested area. The method for determining the direction of congestion is the same as steps 104 and 110 above, and will not be repeated here.

[0083] Step 209: Determine congestion deviation in response to the congestion direction.

[0084] Crowding deviation refers to the numerical value used to show the deviation of the movement of people in a crowded area from the direction of crowding. The calculation method for crowding deviation is the same as step 203 above, and will not be repeated here.

[0085] Step 210: Determine the reverse position based on the congestion deviation.

[0086] The reverse position refers to the location information of people whose congestion deviates from the threshold. The method for determining the reverse position is selected by the staff according to the actual situation, and will not be elaborated here.

[0087] Step 211: Determine the attenuation coefficient based on the reverse position.

[0088] The attenuation coefficient is a numerical value that shows the attenuation of scene sound received from the retrograde position. The farther the retrograde position is from the microphone, the greater the attenuation coefficient. The attenuation coefficient corresponding to the retrograde position can be found in the attenuation correspondence table, which is a data table that records different retrograde positions and their corresponding attenuation coefficients.

[0089] Step 212: Determine the attenuation characteristics by combining the attenuation coefficient and the preset fall characteristics.

[0090] Fall characteristics refer to the sound information that people are likely to make when they fall, such as the sound of a person colliding with an object / ground. Fall characteristics are selected by staff based on the actual situation and will not be elaborated here.

[0091] The attenuation feature is the sound information transmitted from the reverse position to the microphone. The method for determining the attenuation feature is common knowledge to those in the field and will not be elaborated here.

[0092] Step 213: If there is attenuation in the scene sound, define the reverse position as the fall position.

[0093] When people are in a chaotic situation, the system identifies the crowded areas where people are gathered and the direction of movement within those areas. By comparing the direction of the crowd with the direction of movement, it identifies people moving in the wrong direction within the crowded areas. Combined with sound information, it determines whether there have been any falls, thereby improving the accuracy of identifying abnormal behavior.

[0094] Fall detection methods also include: Step 214: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the number of people in the crowd and the edge of the crowd based on the crowded area.

[0095] The number of people in a crowded area refers to the number of people in the crowded area, while the edge of the crowded area refers to the edge location information of the crowded area. The number of people in a crowded area and the edge of the crowded area can be determined by image recognition technology. The methods for determining the number of people in a crowded area and the edge of the crowded area are common knowledge in the field and will not be elaborated here.

[0096] Step 215: Determine the number of people entering and exiting in response to the congestion edge.

[0097] The number of people entering and exiting refers to the number of people entering and exiting a crowded area. It is calculated as the difference between the number of people entering the crowded area and the number of people leaving the crowded area. The number of people entering and exiting can be determined by image recognition technology. The method for determining the number of people entering and exiting is common knowledge in the field and will not be elaborated here.

[0098] Step 216: Calculate the sum of the preset periodic number of people and the number of people entering and exiting, and define it as the real-time number of people.

[0099] The periodic number refers to the number of people in congestion detected in the previous period. The method for retrieving the periodic number is selected by the staff according to the actual situation, and will not be elaborated here.

[0100] The real-time number of people is the number of people that should be in the congested area at this time. The calculation method for the real-time number of people is common knowledge in this field and will not be elaborated here.

[0101] Step 217: If the real-time number of people is greater than the number of people in the crowd, determine the trajectory of people based on the crowded area.

[0102] A real-time number of people greater than the number of people in a crowded area indicates that fewer people are being identified from the scene image at this time, meaning that some people are being occluded. The trajectory of people refers to the movement route of people in the crowded area, which can be determined by image recognition technology. The method for determining the trajectory of people is common knowledge in the field and will not be elaborated here.

[0103] Step 218: Identify the avoidance location from the personnel trajectory.

[0104] The avoidance position is the location where the pedestrian's trajectory should be avoided. The method for determining the avoidance position is common knowledge among those in this field and will not be elaborated here.

[0105] Step 219: Define the avoidance position as the fall position.

[0106] When people gather, the system identifies the number of people leaving and joining the crowded area from the image, thereby calculating the real-time number of people in the crowded area. It also identifies the actual number of people in the crowd from the image. When the real-time number of people exceeds the number of people in the crowd, it is determined that someone has fallen and is being blocked by other people. At this time, the system compares the trajectory of other people in the crowded area to identify the avoidance position where there is a situation, thereby determining that there is a person who has fallen at the avoidance position, thus improving the accuracy of abnormal behavior identification.

[0107] Reference Figure 3 Fall treatment methods include: Step 300: Identify the work distribution from the scene image.

[0108] Work distribution refers to the location information of workers in the target scene. Work distribution can be determined by image recognition technology. The method for determining work distribution is common knowledge in the field and will not be elaborated here.

[0109] Step 301: Select an assistance location from the work distribution based on the fall location.

[0110] The aid location refers to the location information closest to the fall location in the work distribution. The method for determining the aid location is common knowledge among those in the field and will not be elaborated here.

[0111] Step 302: Generate assistance information in response to the assistance location and the fall location, and determine the assistance number based on the assistance location.

[0112] Assistance information refers to route planning information from the assistance location to the fall location. The method for generating assistance information is common knowledge to those in the field and will not be elaborated here.

[0113] The assistance number refers to the identification number of the staff member at the assistance location. The assistance number can be determined by checking the patrol routes of different staff members. The method for determining the assistance number is selected by the staff member based on the actual situation, and will not be elaborated here.

[0114] Step 303: Send the assistance information according to the assistance number to notify staff to assist the person who has fallen.

[0115] When a person falls, the system identifies staff members in the target scene, selects the staff member closest to the fall location for assistance, plans a route from the assistance location to the fall location, and sends it to the staff member at the assistance location, thereby improving the efficiency of assisting the fallen person.

[0116] Fall treatment methods also include: Step 304: Determine the assistance distance based on the assistance location and the fall location.

[0117] The assistance distance refers to the distance between the assistance location and the fall location, that is, the distance between the staff and the person who has fallen. The calculation method of the assistance distance is common knowledge in this field and will not be elaborated here.

[0118] Step 305: When the assistance distance is greater than a preset rapid threshold, the approach flow rate is determined by combining the movement vector and the fall location, and the fall capacity is determined based on the fall location.

[0119] The quick threshold refers to the maximum assistance distance that staff can quickly reach the fall location. The quick threshold is selected by staff based on the actual situation and will not be elaborated here. An assistance distance greater than the quick threshold means that staff cannot quickly assist the fallen person. In this case, the fallen person is more likely to be interfered with by other crowding people, leading to secondary injuries. Approach flow refers to the number of people passing the fall location per unit time. The method for determining approach flow is the same as step 113 above and will not be elaborated here.

[0120] Fall capacity refers to the maximum number of people that the space where a fall occurs can accommodate. You can look up the fall capacity corresponding to a fall location in the capacity mapping table.

[0121] Step 306: Calculate the quotient of the proximity flow and the fall capacity, and define it as the congestion coefficient.

[0122] The crowding coefficient is a numerical value used to show the degree of crowding at the fall location. The more crowded the fall location, the higher the crowding coefficient. The calculation method of the crowding coefficient is common knowledge to those in the field and will not be elaborated here.

[0123] Step 307: If the congestion coefficient is greater than the preset interference threshold, determine the proximity address based on the proximity flow.

[0124] The interference threshold refers to the minimum crowding coefficient at which a person passing through the fall location is likely to interfere with the person who fell. In other words, it's the minimum crowding coefficient at which the fall location is most likely to cause secondary injury. The interference threshold is selected by staff based on the actual situation and will not be elaborated here. A crowding coefficient greater than the interference threshold indicates that the fall location is relatively crowded, and people are more likely to suffer secondary injury. The proximity address refers to the space that a person passing through the fall location would head towards. The method for determining the proximity address can refer to step 111 above and will not be elaborated here.

[0125] Step 308: Determine a detour route in response to the fall location and proximity address.

[0126] A detour route refers to a route that bypasses the location of the fall and goes to a nearby address. The detour route can be determined by referring to the spatial distribution around the target scene. The method for determining the detour route is common knowledge to those in the field and will not be elaborated here.

[0127] Step 309: Generate and display address detour information based on the detour route.

[0128] Address detour information is information that shows detour routes to people in the target scene. The method for generating address detour information is common knowledge to those in the field and will not be elaborated here.

[0129] When the aid location is far from the fall location, the system identifies the approaching flow near the fall location. When the approaching flow is high, it indicates a risk of stampede. At this point, the system identifies the approaching addresses of people near the fall location based on their movement and plans a detour route around the fall location to reach the approaching address. This reduces the risk of secondary injuries to those who have fallen due to excessive crowds at the fall location.

[0130] Fall treatment methods also include: Step 310: If the congestion coefficient is greater than the preset interference threshold, retrieve the historical trajectory based on the proximity flow.

[0131] Historical trajectory refers to the movement route of personnel who have passed through the nearby location, that is, the movement route from the moment they enter the target scene. The method for retrieving historical trajectory is selected by the staff according to the actual situation, and will not be elaborated here.

[0132] Step 311: Identify the source proportion from the historical trajectory.

[0133] The source proportion refers to the proportion of people who entered the target scene from different directions in the historical trajectory. The method for determining the source proportion is common knowledge in this field and will not be elaborated here.

[0134] Step 312: Determine the primary source in response to the source percentage.

[0135] The primary source refers to the direction with the highest proportion of sources. The method for determining the primary source is common knowledge among those in the field and will not be elaborated here.

[0136] Step 313: Determine the access equipment based on the scene location and main source.

[0137] Access control equipment refers to devices such as turnstiles and electric gates that control the number of people entering the target scene from the main source. The source address can be found by combining the scene location and the main source, and then the access control equipment corresponding to the source address and scene location can be found from the equipment correspondence table. The method for determining the source address is the same as step 111 above. The equipment correspondence table is a data table that records different source addresses and scene locations and their corresponding access control equipment.

[0138] Step 314: Generate a passage control signal based on the passage equipment and fall capacity.

[0139] The access control signal is a signal used by communication equipment to control the number of people entering the target scene according to the fall capacity. The method for generating the access control signal is common knowledge to those in the field and will not be elaborated here.

[0140] Step 315: Control the passage device according to the passage control signal.

[0141] When there is a large flow of people, the main source of people approaching the fall location is identified based on the movement of people. Then, passage equipment is matched to control the passage between the main source and the fall location. The flow of people is controlled by the passage equipment according to the carrying capacity of the fall location to reduce the possibility of secondary injury caused by too many people at the fall location.

[0142] Based on the same inventive concept, embodiments of the present invention provide an abnormal behavior recognition system based on the fusion of sound and image information, comprising: The acquisition module is used to acquire scene images and scene sounds; The memory is used to store the program for any of the above-mentioned abnormal behavior recognition methods based on the fusion of sound and image information; The processor is the unit of memory that allows programs to be loaded and executed by the processor.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0144] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. An abnormal behavior recognition method based on the fusion of sound and image information, characterized in that, include: Step 100: Acquire scene images; Step 101: Identify the number of people from the scene image; Step 102: When the number of people is greater than 1, identify the distribution of people from the scene image; Step 103: Determine the movement vector based on the personnel distribution; Step 104: Determine the orientation dispersion in response to the movement vector; Step 105: If the directional dispersion is greater than a preset disorder threshold, collect scene sound; Step 106: Identify the high-pitched sound source from the scene sounds; Step 107: Calculate the quotient of the high-pitched sound source and the number of people, and define it as the high-pitched percentage; Step 108: When the proportion of high-pitched sounds is greater than a preset noise threshold, generate and display an abnormal behavior prompt based on the proportion of high-pitched sounds.

2. The abnormal behavior recognition method based on sound and image fusion information according to claim 1, characterized in that, Also includes: Step 109: If the directional dispersion is not greater than a preset disorder threshold, determine the main direction in response to the movement vector, and determine the scene position based on the scene image; Step 110: Determine the main address based on the scene location and main direction; Step 111: Determine the carrying capacity in response to the primary address; Step 112: Determine the baseline flow rate based on the carrying capacity, and determine the moving flow rate based on the moving vector; Step 113: When the moving flow rate is greater than the reference flow rate, determine the flow retardation coefficient by combining the moving flow rate and the reference flow rate; Step 114: In response to the generation of the flow slowing coefficient, display a flow slowing prompt for personnel.

3. The abnormal behavior recognition method based on sound and image fusion information according to claim 2, characterized in that, It also includes a fall detection method, which includes: Step 200: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the personnel density based on the personnel distribution; Step 201: Determine the isolated location in response to the population density; Step 202: Determine the isolated vector based on the isolated location; Step 203: Determine the degree of deviation by combining the isolated vector and the main direction; Step 204: If the deviation is greater than a preset deviation threshold, identify the person's posture from the scene image based on the isolated position; Step 205: Determine the fall location in response to the person's posture and a preset fall posture; Step 206: Generate and display a fall notification based on the fall location.

4. The abnormal behavior recognition method based on sound and image fusion information according to claim 3, characterized in that, The fall detection method also includes: Step 207: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the crowded area based on the population density; Step 208: Determine the congestion direction based on the congested area; Step 209: Determine congestion deviation in response to the congestion direction; Step 210: Determine the reverse position based on the congestion deviation; Step 211: Determine the attenuation coefficient based on the reverse position; Step 212: Determine the attenuation characteristics by combining the attenuation coefficient and the preset fall characteristics; Step 213: If there is attenuation in the scene sound, define the reverse position as the fall position.

5. The abnormal behavior recognition method based on sound and image fusion information according to claim 4, characterized in that, The fall detection method also includes: Step 214: When the proportion of high-pitched sounds is greater than a preset noise threshold, determine the number of people in the crowd and the edge of the crowd based on the crowded area; Step 215: Determine the number of people entering and exiting in response to the congestion edge; Step 216: Calculate the sum of the preset periodic number of people and the number of people entering and exiting, and define it as the real-time number of people; Step 217: If the real-time number of people is greater than the number of people in the crowd, determine the trajectory of people based on the crowded area; Step 218: Identify the avoidance location from the personnel trajectory; Step 219: Define the avoidance position as the fall position.

6. The abnormal behavior recognition method based on sound and image fusion information according to claim 5, characterized in that, It also includes a fall handling method, which includes: Step 300: Identify the work distribution from the scene image; Step 301: Select an assistance location from the work distribution based on the fall location; Step 302: Generate assistance information in response to the assistance location and the fall location, and determine the assistance number based on the assistance location; Step 303: Send the assistance information according to the assistance number to notify staff to assist the person who has fallen.

7. The abnormal behavior recognition method based on sound and image fusion information according to claim 6, characterized in that, The fall handling method also includes: Step 304: Determine the assistance distance based on the assistance location and the fall location; Step 305: When the assistance distance is greater than a preset rapid threshold, the approach flow rate is determined by combining the movement vector and the fall location, and the fall capacity is determined based on the fall location; Step 306: Calculate the quotient of the proximity flow and the fall capacity, and define it as the congestion coefficient; Step 307: If the congestion coefficient is greater than the preset interference threshold, determine the proximity address based on the proximity flow. Step 308: Determine a detour route in response to the fall location and proximity address; Step 309: Generate and display address detour information based on the detour route.

8. The abnormal behavior recognition method based on sound and image fusion information according to claim 7, characterized in that, The fall handling method also includes: Step 310: If the congestion coefficient is greater than the preset interference threshold, retrieve the historical trajectory based on the proximity flow. Step 311: Identify the source proportion from the historical trajectory; Step 312: Determine the primary source in response to the stated source percentage; Step 313: Determine the access equipment based on the scene location and main source; Step 314: Generate a passage control signal based on the passage equipment and fall detection capacity; Step 315: Control the passage device according to the passage control signal.

9. An abnormal behavior recognition system based on the fusion of sound and image information, characterized in that, include: The acquisition module is used to acquire scene images and scene sounds; A memory for storing a program for an abnormal behavior recognition method based on sound and image fusion information as described in any one of claims 1 to 8; The processor is the unit of memory that allows programs to be loaded and executed by the processor.

Images