Multi-target fusion dangerous behavior recognition algorithm
Through the multi-objective fusion hazard behavior recognition algorithm, combining the identity recognition model of dangerous objects and personnel and the human body key point detection model, a spatial vector model is constructed, which solves the problem of police responding to the safety risks of blind spots in a complex law enforcement environment, and achieves the risk of danger Real-time identification and prediction of items and behaviors improves the recognition capabilities and response speed of the monitoring system.
Patent Information
- Application Number
- CN202411182363.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-03
AI Technical Summary
In a complex law enforcement environment, how police officers can effectively deal with the safety risks brought by blind spots in their vision, the existing technology is too single, and it is difficult to fully identify and deal with various uncontrollable risk factors.
Using a multi-objective fusion hazardous behavior recognition algorithm, the surveillance image is captured through high-definition cameras, combined with the identity recognition model of dangerous objects and personnel and the detection model of human key points, a spatial vector model is constructed to conduct overall and local parts of individual behavior in the monitoring scenario. Comprehensive analysis to achieve real-time prediction and visualization.
It significantly improves the identification ability and response speed of the monitoring system in complex environments, can promptly identify dangerous items and their dangerous actions in the surveillance screen and helps police effectively deal with blind spots in the field of vision and other uncontrollable risks, and improves safety monitoring. accuracy and efficiency.
Smart Images

Figure CN119068550A_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the field of computer vision technology, and specifically to a dangerous behavior recognition algorithm with multi-target fusion. Background Art
[0002] In a complex law enforcement environment, how police officers can effectively deal with various uncontrollable risk factors, especially the safety risks brought by blind spots, has become an urgent problem to be solved. Summary of the invention
[0003] The technical solution of the present invention aims at the technical problem that the existing technical solutions are too single, and provides a solution that is significantly different from the existing technologies. It mainly provides a multi-target fusion dangerous behavior recognition algorithm to solve the technical problem raised in the above background technology on how police officers can effectively deal with various uncontrollable risk factors, especially the safety risks caused by blind spots in vision.
[0004] The technical solution adopted by the present invention to solve the above technical problems is:
[0005] A multi-target fusion dangerous behavior recognition algorithm includes the following steps:
[0006] S1, data acquisition and preprocessing: use high-definition cameras to capture real-time monitoring images and synchronously transmit video data streams to the back-end processing system;
[0007] S2, model parallel processing: through the dangerous goods and personnel identification model and the human body key point detection model identification processing, the dangerous goods and personnel identification model is used to identify and distinguish dangerous goods and non-dangerous goods, and identify the identity of law enforcement officers; the human body key point detection model is used to identify the key points of the body of the person in the monitoring picture;
[0008] S3, Multi-target fusion and analysis: Through the dangerous behavior recognition algorithm based on spatial vectors, based on the output of the human key point detection model, combined with the recognition results of the dangerous goods and personnel identification models, by constructing a spatial vector model, the overall and local comprehensive analysis of the behavior patterns of individuals in the monitoring scene is carried out;
[0009] S4. Real-time prediction and visualization: Output timely and accurate dangerous behavior prediction results and display them through a visualization interface.
[0010] Further, in step S2, the human body key point detection model includes a hand-raising recognition algorithm, and the hand-raising recognition algorithm includes:
[0011] S2-1-1. Establish a plane rectangular coordinate system for the image to obtain the coordinates of the human body's shoulder, elbow, and wrist;
[0012] S2-1-2, calculate and obtain the shoulder-elbow vector and the elbow-wrist vector;
[0013] S2-1-3, calculate the angle between the shoulder-elbow vector and the elbow-wrist vector, and make a judgment based on the preset threshold to distinguish between "direct provocation" and "bending the arm to accumulate strength";
[0014] S2-1-4. For the "direct provocation" situation, first determine whether the elbow is higher than the shoulder, then determine whether the wrist is higher than the elbow. If the wrist is higher than the elbow and the elbow is higher than the shoulder, the target is determined to raise his hand; otherwise, there is no possibility of raising his hand;
[0015] For the "bent arm to accumulate strength" situation, determine whether the wrist is higher than the shoulder. If the wrist is higher than the shoulder, the target is judged to have raised his hand; otherwise, there is no possibility of raising his hand.
[0016] Further, in step S2, the human body key point detection model includes a fall recognition algorithm, and the fall recognition algorithm includes:
[0017] S2-2-1. Establish a plane rectangular coordinate system for the image to obtain coordinates of two parts with obvious spacing that are difficult for the human body to actively control;
[0018] S2-2-2, obtain the maximum value of the difference between the horizontal coordinates and the maximum value of the difference between the vertical coordinates of the two parts;
[0019] S2-2-3. Compare the maximum value of the horizontal coordinate difference with the maximum value of the vertical coordinate difference. If the maximum value of the horizontal coordinate difference is greater than the maximum value of the vertical coordinate difference, it is preliminarily determined that the detection target may fall; if the maximum value of the horizontal coordinate difference is less than the maximum value of the vertical coordinate difference, it is preliminarily determined that the detection target is standing.
[0020] Furthermore, the fall recognition algorithm also includes:
[0021] S2-2-4, respectively obtain the vectors formed by the two selected parts on the left side of the human body and the two selected parts on the right side of the human body, and compare the moduli to obtain the vector with a larger modulus;
[0022] S2-2-5. First find the unit vector of the vector, then find the angle between the unit vector and the y-axis, and then find the inner product of the vector;
[0023] S2-2-6. Use the Taylor series of the arccosine function to derive the approximate formula for arccosine calculation, and use the formula to calculate the angle between the human body vector and the y-axis;
[0024] S2-2-7. Set threshold θ according to camera angle 0 , if the θ calculated above is less than the threshold θ 0 , it means that the person is in a standing state (including bending over). If the θ calculated above is greater than the threshold θ 0 , it means that the person has fallen.
[0025] Furthermore, step S3 includes a target and limb pairing algorithm based on Manhattan distance and a dangerous action prediction algorithm based on Taylor series expansion.
[0026] Furthermore, the target and limb pairing algorithm based on Manhattan distance includes:
[0027] S3-1-1. Obtain the frame coordinates of the dangerous goods and the coordinates of the wrists of the personnel through the dangerous goods and personnel identification model;
[0028] S3-1-2. The coordinates of the center point of the dangerous item are obtained according to the frame coordinates of the dangerous item, and then the Manhattan distances from the center point of the dangerous item to the left and right hands are calculated respectively. If the Manhattan distance to the left hand is smaller, it is determined that the dangerous item is held in the left hand, otherwise it is determined that the dangerous item is held in the right hand.
[0029] Furthermore, the dangerous action prediction algorithm based on Taylor series expansion includes:
[0030] S3-2-1, obtain the shoulder-elbow vector and elbow-wrist vector through the human body key point detection model recognition;
[0031] S3-2-2. For the shoulder-elbow vector and the elbow-wrist vector, calculate their Euclidean norms respectively;
[0032] S3-2-3, according to the vector and the vector modulus, obtain the unit vector;
[0033] S3-2-4. Calculate the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector, and compare them with the threshold. If it is less than the threshold, the arm is bent and there is a possibility of throwing or slashing; if it is greater than the threshold, the possibility of throwing or slashing is small.
[0034] Furthermore, in step S3-2-4, a multi-frame prediction mechanism is adopted, and the formula is:
[0035]
[0036] in, Represents the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of n frames, α 11 ⊙α 21 This is the calculation formula for the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of the first frame.
[0037] Compared with the prior art, the present invention has the following beneficial effects:
[0038] (1) The present invention can timely and effectively identify dangerous objects and dangerous actions of their holders in the monitoring screen through steps such as data collection and preprocessing, model parallel processing, multi-target fusion and analysis, real-time prediction and visualization, so that security monitoring personnel can quickly identify and respond to potential security threats, and help civilian and auxiliary police effectively deal with various uncontrollable risk factors, especially the security risks caused by blind spots.
[0039] (2) The present invention processes the monitoring screen through the dangerous goods and personnel identification model and the human body key point detection model. The dangerous goods and personnel identification model is used to identify and distinguish dangerous goods from non-dangerous goods, and at the same time identify the identity of law enforcement personnel; the human body key point detection model is used to identify the key points of the body of the person in the monitoring screen. The target recognition model is used to achieve information compensation for the human body key point detection. When these two models are run independently, their respective limitations limit their effectiveness in practical applications. Using these two models at the same time can effectively break through their respective limitations and improve recognition capabilities.
[0040] (3) The dangerous behavior recognition algorithm based on spatial vectors of the present invention combines the recognition results of the dangerous goods and personnel identification model and the output of the human key point detection model, and constructs a spatial vector model to conduct a comprehensive analysis of the overall and local behavior patterns of individuals in the monitoring scene. Through this method of multi-target fusion and spatial vector analysis, the recognition ability and response speed of the monitoring system in complex environments are significantly improved, bringing a breakthrough technological progress to the field of intelligent security monitoring.
[0041] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of a police law enforcement safety guarantee system in a multi-risk environment in an embodiment of the present invention, wherein: Figure 1 (a) is the scene graph applicable to the system. Figure 1 (b) is the system workflow diagram;
[0043] Figure 2 A flowchart of a multi-target fusion dangerous behavior recognition algorithm provided by the present invention;
[0044] Figure 3 This is a diagram showing the operation effect of hand raising recognition in an embodiment of the present invention, wherein: Figure 3 (a) is an effective recognition diagram of raising the right hand. Figure 3 (b) is a valid recognition diagram of raising the left hand;
[0045] Figure 4 A schematic diagram of establishing a rectangular coordinate system during hand-raising recognition in an embodiment of the present invention;
[0046] Figure 5 is a picture of a target raising his hand used in an embodiment of the present invention;
[0047] Figure 6 Schematic diagram of analyzing key points of a human body in an embodiment of the present invention, wherein: Figure 6 (a) is the first hand-raising situation. Figure 6 (b) is the second hand-raising situation;
[0048] Figure 7 A schematic diagram of a multi-target determination and recognition target having a hand-raising action in an embodiment of the present invention;
[0049] Figure 8 is a schematic diagram of an example of fall recognition in an embodiment of the present invention;
[0050] Fig. 9 A schematic diagram of extracting the target eye and ankle coordinates in an embodiment of the present invention;
[0051] Fig.10 A schematic diagram of obtaining the angle between the human body vector and the y-axis in an embodiment of the present invention;
[0052] Fig.11 is a schematic diagram of recognition of an example including falling and hand-raising actions in an embodiment of the present invention;
[0053] Fig.12 This is a schematic diagram of identification in a multi-person situation in an embodiment of the present invention;
[0054] Fig.13 This is an actual use recognition diagram of fall recognition in an embodiment of the present invention;
[0055] Fig.14 Schematic diagram of using the dangerous object and person identification model to obtain the coordinates of the dangerous object and the person's hand in an embodiment of the present invention, wherein: Fig.14 (a) is the tool coordinate, Fig.14 (b) is the coordinates of the person’s hand;
[0056] Fig.15 Schematic diagram of the analysis of the dangerous object and limb pairing process in an embodiment of the present invention;
[0057] Fig.16 This is a result diagram of dangerous objects and limb pairing in an embodiment of the present invention;
[0058] Fig.17 A schematic diagram of obtaining a right shoulder-elbow vector and a right elbow-wrist vector in an embodiment of the present invention;
[0059] Fig.18 This is a schematic diagram of outputting the calculation result in the upper left corner of the screen in an embodiment of the present invention. DETAILED DESCRIPTION
[0060] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.
[0061] Design a police law enforcement safety guarantee system in a multi-risk environment. The system is applicable to scenarios such as Figure 1 (a) is shown. It is mainly used to identify the behavior of police officers at law enforcement sites, allowing police officers to avoid risks including blind spots in their field of vision in a timely manner.
[0062] The system workflow is as follows Figure 1 (b) as shown, including:
[0063] 1) Law enforcement recorders, cameras and other equipment capture real-time monitoring images;
[0064] 2) Generate video data stream;
[0065] 3) Target detection and key point detection are performed separately. Target detection uses the dangerous goods and personnel identification model, and key point detection uses the human key point detection model;
[0066] 4) The results of target detection and key point detection are transmitted to the multi-target fusion and dangerous behavior recognition algorithms based on spatial vectors respectively;
[0067] 5) Obtain the calculation results and transmit them to the IoT cloud platform. The command center monitors the IoT cloud platform;
[0068] 6) The IoT cloud platform determines whether there is any person engaging in dangerous behavior. If so, an alarm signal is sent to the alarm device; if not, it returns to step 2).
[0069] Please refer to the attached Figure 2 , a multi-objective fusion dangerous behavior recognition algorithm, including:
[0070] 1) Data collection and preprocessing
[0071] Use high-definition cameras to capture real-time monitoring images and synchronously transmit video data streams to the back-end processing system.
[0072] 2) Model parallel processing
[0073] The video data stream is fed into two specially designed sub-models - the dangerous goods and personnel identification model and the human body key point detection model. The former is responsible for identifying and distinguishing dangerous goods (such as controlled knives, etc.) and non-dangerous goods (such as mobile phones) from the surveillance images, and identifying the identity of law enforcement officers, and transmitting the identified object location information and personnel identity information to the pre-allocated memory space; the latter uses advanced computer vision technology to identify the key points of the person's body in the surveillance image, and also transmits the key point data to the designated memory space.
[0074] Information compensation of the target recognition model for key point detection of the human body: In the field of security monitoring, the application of the key point detection model of the human body and the identification model of dangerous goods and personnel is crucial to improving the effectiveness of the monitoring system. However, when these two models are run independently, their respective limitations limit their effectiveness in practical applications. Specifically, when the target recognition model is executed independently, its function is limited to identifying the identity of the target individual and the objects held by him. However, the model lacks the ability to dynamically analyze specific behaviors such as holding a knife, and cannot accurately determine whether the individual is performing dangerous actions. On the other hand, when the key point detection model of the human body is run alone, its main function is to capture and analyze the movement state of the human body, but it cannot combine these movements with the objects held by the hands, so that it is impossible to effectively judge whether the individual is performing dangerous behaviors. Therefore, these two models have certain functional limitations when running independently, and it is difficult to comprehensively evaluate the potential risks of individual behavior. The present invention can achieve mutual information compensation by adopting these two models at the same time, which is conducive to comprehensively evaluating the potential risks of individual behavior.
[0075] 3) Multi-target fusion and analysis
[0076] After collecting the output results of the two sub-models, the core of this algorithm, the dangerous behavior recognition algorithm based on spatial vectors, begins to operate. Based on the output of the human key point detection model, the algorithm combines the recognition results of the dangerous goods and personnel identification models, and constructs a spatial vector model to conduct a comprehensive analysis of the overall and local behavior patterns of individuals in the monitoring scene. This process involves complex data processing and pattern recognition technologies, including but not limited to target tracking, behavior pattern matching, and abnormal behavior detection.
[0077] 4) Real-time prediction and visualization
[0078] Ultimately, the algorithm outputs timely and accurate prediction results of dangerous behaviors and displays them through a visual interface, allowing security monitoring personnel to quickly identify and respond to potential security threats.
[0079] Through this method of multi-target fusion and spatial vector analysis, this algorithm significantly improves the recognition ability and response speed of the monitoring system in complex environments, bringing breakthrough technological progress to the field of intelligent security monitoring.
[0080] 1. Human Behavior Recognition Algorithm
[0081] The algorithm only uses the human key point recognition model for calculation. The algorithm first identifies the key points of the human body (including eyes, nose, ears, shoulders, elbows, wrists, and feet), then obtains the coordinates of each key point of the human body, and finally determines the target behavior based on the coordinate relationship of each key point.
[0082] 1.1. Hand Raising Recognition
[0083] When the custom gesture recognition model based on space vectors calculates the target's hand, the image will be mirrored and flipped. The running effect is as follows: Figure 3 (a) and 3(b).
[0084] Establish a rectangular coordinate system: Establish a plane rectangular coordinate system for the image, with the horizontal axis being the x-axis and the vertical axis being the y-axis. Here, the upper left corner of the image is the origin of the coordinate system. The x value increases to the right along the x-axis, and the y value increases downward along the y-axis (contrary to the commonly used mathematical plane rectangular coordinate system), such as Figure 4 shown.
[0085] Then use a picture of the target raising his hand for analysis, such as Figure 5 shown.
[0086] In order to facilitate the display of the algorithm and eliminate interference factors, the key points of the human body are analyzed, such as Figure 6 (a) and (b), where Figure 6 (a) is the first type of hand-raising situation, corresponding to the situation of “direct provocation”; Figure 6 (b) is the second hand-raising situation, corresponding to the "bent arm to accumulate strength" situation. These two situations are distinguished according to the degree of arm bending. The judgment method is: whether the angle between the forearm and the upper arm (that is, the angle between the shoulder-elbow vector and the elbow-wrist vector) exceeds 90 degrees or other set thresholds. If it exceeds the threshold, there is a possibility of "direct provocation", otherwise there is a possibility of "bent arm to accumulate strength".
[0087] For the first hand-raising situation (taking the left hand as an example), the shoulder coordinates LeftShoulder (ls_x, ls_y), the elbow coordinates LeftElbow (le_x, le_y), and the wrist coordinates LeftWrist (lw_x, lw_y) are used to obtain the shoulder-elbow vector LSE (le_x-ls_x, le_y-ls_y). If the y direction of the vector points to the x-axis, that is, le_y-ls_y<0, then the target may have raised its hand; continue to calculate the elbow-wrist vector LEW (lw_x-le_x, lw_y-le_y). If the y direction of the vector points to the x-axis, that is, lw_y-le_y<0, then it can be further inferred that the target may have raised its hand; finally, confirm the vertical coordinates of the wrist, elbow, and shoulder. If the comparison result is lw_y<le_y<ls_y, it can be determined that the target is in the first hand-raising situation.
[0088] For the second hand-raising situation, the shoulder coordinates RightShoulder (rs_x, rs_y), the elbow coordinates RightElbow (re_x, re_y), and the wrist coordinates RightWrist (rw_x, rw_y) are used to obtain the shoulder-elbow vector RSE (re_x-rs_x, re_y-rs_y). It is not possible to tell whether the target has raised its hands from the shoulder-elbow vector alone, so the elbow-wrist vector is calculated to obtain REW (rw_x-re_x, rw_y-re_y) to prepare for the judgment of the subsequent swinging and charging action. Since this algorithm is dedicated to identifying dangerous behaviors, the action of "raising hands" is defined as the wrist exceeding the shoulder. Therefore, if rs_y>rw_y, it is preliminarily determined that the identified target has raised its hands, such as Figure 7 shown.
[0089] Hand-raising recognition is used in the following text for integration with dangerous goods and personnel identification models.
[0090] 1.2. Fall Detection
[0091] When it is recognized that a person has fallen and there are no police officers at the scene, the system will automatically send an alarm to notify the command center for assessment and processing, and the command center will decide whether to send police; when it is recognized that a police officer has fallen, the system will also send an alarm to notify the command center, and the command center will assess the situation at the scene and decide whether additional police force is needed for support.
[0092] After establishing the rectangular coordinate system, obtain and calculate the coordinates of the key points of the target body. There are multiple detection targets in the sample. First, extract the fallen detection object and identify it, such as Figure 8 shown.
[0093] In the field of human posture analysis, identifying whether an individual has fallen is one of the key tasks. In order to accurately detect falling behavior, the algorithm needs to comprehensively consider the geometric distribution of the human body in space and its dynamic changes. Specifically, there are significant differences in the representation of the human body posture in the coordinate system between the upright state and the fallen state: when standing upright, the vertex coordinates of the human body (the position information of the highest and lowest points of the human body in three-dimensional space) are roughly parallel to the y-axis, while when falling, they are parallel to the x-axis. In addition, the height of a person is usually greater than his width. This biological characteristic provides an important reference for fall detection.
[0094] During a fall, an individual often experiences a moment of loss of balance, accompanied by rapid waving of the arms in an attempt to regain balance. Given the high speed and irregularity of arm movements, algorithms often ignore arm position information during fall detection, and instead focus on body parts that are difficult for individuals to actively control, such as the eyes, ears, legs, and feet. This algorithm specifically selects the eyes and feet as key calculation points, because these parts are relatively stable during a fall event, and have a high degree of discrimination and reliability for determining the state of a fall. By comparing the changes in the coordinates of these key points in the upright state and the fall state, combined with time series analysis, the algorithm can effectively identify fall events. Therefore, this embodiment chooses to extract the target eye and ankle coordinates, such as Fig. 9 shown.
[0095] Find the maximum value of the difference between the vertical coordinates of the eyes and feet:
[0096] a_y=max(|ra_y-re_y|,|ra_y-le_y|,|la_y-re_y|,|la_y-le_y|) (3-1)
[0097] Wherein, |ra_y-re_y| represents the absolute value of the difference between the right eye and the right ankle, |ra_y-le_y| represents the absolute value of the difference between the left eye and the right ankle, |la_y-re_y| represents the absolute value of the difference between the right eye and the left ankle, and |la_y-le_y| represents the absolute value of the difference between the left eye and the left ankle;
[0098] Find the maximum value of the difference between the horizontal coordinates of the eyes and feet:
[0099] a_x=max(|ra_x-re_x|,|ra_x-le_x|,|la_x-re_x|,|la_x-le_x|) (3-2)
[0100] Wherein, |ra_x-re_x| represents the absolute value of the horizontal coordinate difference between the right eye and the right ankle, |ra_x-le_x| represents the absolute value of the horizontal coordinate difference between the left eye and the right ankle, |la_x-re_x| represents the absolute value of the horizontal coordinate difference between the right eye and the left ankle, and |la_x-le_x| represents the absolute value of the horizontal coordinate difference between the left eye and the left ankle;
[0101] If α_x<α_y, it means that the detection target is most likely standing, and if α_x>α_y, the detection target is likely to fall.
[0102] In order to further improve the accuracy of judgment, inverse trigonometric function calculation is introduced here to calibrate the calculation results. First, the vectors formed by the left eye and left foot and the right eye and right foot are obtained respectively, and the vector with a larger modulus is obtained by comparing the modulus. The modulus of the vector formed by the left eye and left foot is shown in formula (3-3), and the modulus of the vector formed by the right eye and right foot is shown in formula (3-4).
[0103]
[0104] Find the vector Find the unit vector of the α vector Known α 0 Vector, find α 0 The angle between the vector and the y-axis, here we introduce the unit vector γ 0 =(0,1), use the vector inner product to get σ=α 0 ⊙γ 0 .
[0105] The above calculation results are preliminarily calibrated through σ: it is known that the inner product of the human body vector and the y-axis vector is σ, that is, the cosine value of the angle between the human body vector and the y-axis is σ. Therefore, when σ is greater than the threshold, it means that the angle between the human body vector and the y-axis is small, indicating that the human body is in a standing state; when σ is less than the threshold, it means that the angle between the human body vector and the y-axis is large, indicating that the human body is in a falling state.
[0106] Here, the Taylor series expansion of the arccosine function is shown in formula (3-5), and function (3-7) is used to derive the angle between the human body vector and the y-axis (the angle between the human body vector and the y-axis (fully upright state) can be obtained by converting the human body vector / human body vector modulus to the unit vector of the human body vector, and the inner product with the (0, 1) vector and using the Taylor series expansion of the arccosine function), as shown in Fig.10 shown.
[0107]
[0109] Known
[0110]
[0111] Among them, x represents the angle between the human body vector and the y-axis, which is used to determine whether the human body is in a standing state. n represents the number of formula items required according to user needs. The larger n is, the closer it is to the exact value.
[0112] In order to balance computational efficiency and angle accuracy, this algorithm takes the first ten terms of the Taylor series expansion of the inverse cosine function to calculate the angle.
[0113] According to the threshold set by the user, if the angle between the human body vector and the y-axis is greater than the threshold, it indicates falling, and if the angle is less than the threshold, it indicates standing.
[0114] Combined with the hand-raising algorithm, the football player in the example is judged to have an overall action of falling and a local action of raising his hand during detection, such as Fig.11 As shown in the figure; in the case of multiple people, the player's body movements can still be accurately identified, such as Fig.12 In actual use, the model recognition effect is as follows Fig.13 shown.
[0115] 2. Multi-target classification and dangerous behavior recognition fusion algorithm based on spatial vector
[0116] 2.1. Manhattan distance-based target and limb pairing algorithm
[0117] Since a single model has limitations in the accuracy of predicting the target's dangerous behavior, in order to overcome this difficulty, the algorithm proposed in this invention adopts a multi-target data fusion strategy and combines it with a high-risk behavior recognition technology based on spatial vectors.
[0118] The algorithm design allows the dangerous goods and personnel identification models and the human key point detection model to integrate their independent data into the algorithm's preset memory structure for comprehensive analysis and processing after obtaining them. In the algorithm's execution process, the initial step focuses on solving the problem of individual identity confirmation, that is, determining which specific individual holds the dangerous goods and which part of the individual (such as which hand) holds the goods. This step is crucial for subsequent behavioral analysis because it provides the algorithm with precise contextual information, making behavioral predictions more accurate and reliable.
[0119] 1) Coordinate acquisition
[0120] Taking a knife as an example, the coordinates of the dangerous object and the person's hand are obtained using the dangerous object and person identification model. Fig.14 (a) and 14(b).
[0121] 2) Pairing dangerous objects with body parts
[0122] The left and right wrist point coordinates and the dangerous goods frame coordinates (two coordinates, respectively (k 1- x,k 1- y) and (k 2- x,k 2- y)), to simplify the calculation, the coordinates of the center point of the dangerous object are obtained according to the frame coordinates of the dangerous object, and then the Manhattan distances from the center point of the dangerous object to the left and right hands are calculated respectively. If the Manhattan distance to the left hand is small, it is determined that the knife is held by the left hand, otherwise it is determined that the knife is held by the right hand. Fig.15 shown.
[0123] Obtain the coordinates of the center point of the dangerous goods:
[0124]
[0125] The Manhattan distance from the center of the dangerous object to the right hand:
[0126]
[0127] Among them, rw_x represents the horizontal coordinate of the right wrist, and rw_y represents the vertical coordinate of the right wrist;
[0128] The Manhattan distance from the center of the dangerous object to the left hand:
[0129]
[0130] Wherein, lw_x represents the horizontal coordinate of the left wrist, and lw_y represents the vertical coordinate of the left wrist;
[0131] If x 1 <x 2 It is judged that the knife is held in the right hand. Fig.16 as shown; otherwise it is judged as holding the knife with the left hand.
[0132] 2.2. Dangerous action prediction algorithm based on Taylor series expansion
[0133] In order to ensure the personal safety of law enforcement officers and other on-site personnel, after identifying potential dangerous carriers, the present invention proposes a dangerous behavior prediction algorithm based on motion analysis. The algorithm aims to predict whether the target individual is likely to take dangerous behaviors by analyzing their body movements, thereby providing advance action guidance for on-site personnel in emergency situations.
[0134] When dealing with armed threat scenarios, individuals usually first perform a charging action, which is manifested as raising and bending the arm, preparing to throw or slash. In response to this behavioral feature, the algorithm implements frame-by-frame motion detection to capture and analyze potential precursors to attacks. At the same time, combined with the personnel identification model, the algorithm can identify the specific objects in the hands of the target individual. For example, if the model recognizes that the object in the hand is a mobile phone rather than a dangerous object, and the target individual is performing actions similar to making a phone call, the system will determine that his behavior is not aggressive, thereby avoiding triggering false alarm instructions.
[0135] In this way, the algorithm not only improves the accuracy of identifying potentially dangerous behaviors, but also effectively reduces the risk of misjudgment, providing more accurate safety guarantees for on-site disposal.
[0136] 1) Shoulder and elbow vector acquisition
[0137] First, use the human key point detection model to obtain the shoulder-elbow vector and elbow-wrist vector, such as Fig.17 shown.
[0138] 2) Target behavior discrimination algorithm based on vector operation
[0139] Based on the principles of human biomechanics, the individual's action intention is inferred by analyzing the vector relationship between limb joints. Specifically, when it is necessary to obtain their unit vectors from the known shoulder-elbow vector (the vector from the shoulder to the elbow) and elbow-wrist vector (the vector from the elbow to the wrist), the following steps are used:
[0140] (1) Calculate whether the arm has the second hand-raising action mentioned above, which can be simplified to determine whether the wrist is higher than the shoulder;
[0141] (2) For the shoulder-elbow vector and the elbow-wrist vector, calculate their Euclidean norms respectively, as shown in formula (3-11);
[0142]
[0143] (3) According to the vector and the vector modulus, the unit vector is obtained, as shown in equations (3-12) and (3-13);
[0144]
[0145] (4) From the perspective of human physiology, when an individual holds a dangerous object and is ready to throw or chop, he or she usually raises the arm and bends the forearm, forming a sideways throwing posture. In this posture, the directions of the shoulder-elbow vector and the elbow-wrist vector are obviously inconsistent, so their unit vector inner product will be less than the threshold cosθ 0On the contrary, if the individual is making a provocative action or has made a slashing action, the arm will usually be nearly straight. At this time, the directions of the shoulder-elbow vector and the elbow-wrist vector will tend to be consistent, and their unit vector inner product, as shown in formula (3-14), will be greater than the threshold, or even close to 1. At this time, the possibility of throwing or slashing is small.
[0146] cosθ=α 1 ⊙α 2 (3-14)
[0147] If cosθ<cosθ 0 , the arm is bent, and it is possible to throw or slash; if cosθ>cosθ 0 , then throwing or slashing is less likely.
[0148] Since human behavior is accidental and uncertain, there may be errors in the single-frame prediction of the target behavior. However, due to the limitation of human function, generally speaking, human behavior is still stable under the lens of 30-60 frames per second. Therefore, in order to ensure the accuracy of prediction, this algorithm designs a multi-frame prediction mechanism, the formula is:
[0149]
[0150] in, Represents the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of n frames, α 11 ⊙α 21 This is the calculation formula for the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of the first frame.
[0151] Through the Taylor series expansion of the arccosine function, the angle θ between the forearm and the back arm can be obtained:
[0152]
[0153] When θ<θ 0 If the slash is thrown or chopped, the result will be output in the upper left corner of the screen. Fig.18 shown.
[0154] In the example, the judgment result is that there is a possibility of slashing or throwing a knife, and the relevant alarm information is issued.
[0155] The above is an exemplary description of the present invention in combination with the accompanying drawings. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.
Claims
1. A multi-target fusion dangerous behavior recognition algorithm, characterized by: The steps include: S1, data acquisition and preprocessing: use high-definition cameras to capture real-time monitoring images and synchronously transmit video data streams to the back-end processing system; S2, model parallel processing: through the dangerous goods and personnel identification model and the human body key point detection model identification processing, the dangerous goods and personnel identification model is used to identify and distinguish dangerous goods and non-dangerous goods, and identify the identity of law enforcement officers; The human body key point detection model is used to identify the key points of the person's body in the surveillance image; S3, Multi-target fusion and analysis: Through the dangerous behavior recognition algorithm based on spatial vectors, based on the output of the human key point detection model, combined with the recognition results of the dangerous goods and personnel identification models, by constructing a spatial vector model, the overall and local comprehensive analysis of the behavior patterns of individuals in the monitoring scene is carried out; S4. Real-time prediction and visualization: Output timely and accurate dangerous behavior prediction results and display them through a visualization interface.
2. According to the multi-target fusion dangerous behavior recognition algorithm of claim 1, it is characterized by: In step S2, the human key point detection model includes a hand-raising recognition algorithm, and the hand-raising recognition algorithm includes: S2-1-1. Establish a plane rectangular coordinate system for the image to obtain the coordinates of the human body's shoulder, elbow, and wrist; S2-1-2, calculate and obtain the shoulder-elbow vector and the elbow-wrist vector; S2-1-3, calculate the angle between the shoulder-elbow vector and the elbow-wrist vector, and make a judgment based on the preset threshold to distinguish between "direct provocation" and "bending the arm to accumulate strength" situations; S2-1-4, for the "direct provocation" situation, first determine whether the elbow is higher than the shoulder, then determine whether the wrist is higher than the elbow. If the wrist is higher than the elbow and the elbow is higher than the shoulder, the target is determined to raise his hand; otherwise, there is no possibility of raising his hand; For the "bent arm to store power" situation, determine whether the wrist is higher than the shoulder. If the wrist is higher than the shoulder, the target is judged to have raised his hand; otherwise, there is no possibility of raising his hand.
3. The multi-target fusion dangerous behavior recognition algorithm according to claim 1 is characterized by: In step S2, the human key point detection model includes a fall recognition algorithm, and the fall recognition algorithm includes: S2-2-1. Establish a plane rectangular coordinate system for the image to obtain coordinates of two parts with obvious spacing that are difficult for the human body to actively control; S2-2-2, obtain the maximum value of the difference between the horizontal coordinates and the maximum value of the difference between the vertical coordinates of the two parts; S2-2-3. Compare the maximum value of the horizontal coordinate difference with the maximum value of the vertical coordinate difference. If the maximum value of the horizontal coordinate difference is greater than the maximum value of the vertical coordinate difference, it is preliminarily determined that the detection target may fall; if the maximum value of the horizontal coordinate difference is less than the maximum value of the vertical coordinate difference, it is preliminarily determined that the detection target is standing.
4. The multi-target fusion dangerous behavior recognition algorithm according to claim 3 is characterized by: The fall recognition algorithm also includes: S2-2-4, respectively obtain the vectors formed by the two selected parts on the left side of the human body and the two selected parts on the right side of the human body, and compare the moduli to obtain the vector with a larger modulus; S2-2-5. First find the unit vector of the vector, then find the angle between the unit vector and the y-axis, and then find the inner product of the vector; S2-2-6. Use the Taylor series of the arccosine function to derive the approximate formula for arccosine calculation, and use the formula to calculate the angle θ between the human body vector and the y-axis; S2-2-7. Set a threshold value θ0 according to the camera angle. If the θ calculated above is smaller than the threshold value θ0, it indicates that the person is standing. If the θ calculated above is larger than the threshold value θ0, it indicates that the person has fallen.
5. The multi-target fusion dangerous behavior recognition algorithm according to claim 1 is characterized by: Step S3 includes a target and limb pairing algorithm based on Manhattan distance and a dangerous action prediction algorithm based on Taylor series expansion.
6. The multi-target fusion dangerous behavior recognition algorithm according to claim 5 is characterized by: The Manhattan distance-based target and limb pairing algorithm includes: S3-1-1. Obtain the frame coordinates of the dangerous goods and the coordinates of the wrists of the personnel through the dangerous goods and personnel identification model; S3-1-2. The coordinates of the center point of the dangerous item are obtained according to the frame coordinates of the dangerous item, and then the Manhattan distances from the center point of the dangerous item to the left and right hands are calculated respectively. If the Manhattan distance to the left hand is smaller, it is determined that the dangerous item is held by the left hand, otherwise it is determined that the dangerous item is held by the right hand.
7. The multi-target fusion dangerous behavior recognition algorithm according to claim 5 is characterized by: The dangerous action prediction algorithm based on Taylor series expansion includes: S3-2-1, obtain the shoulder-elbow vector and elbow-wrist vector through the human body key point detection model recognition; S3-2-2. For the shoulder-elbow vector and the elbow-wrist vector, calculate their Euclidean norms respectively; S3-2-3, according to the vector and the vector modulus, obtain the unit vector; S3-2-4. Calculate the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector, and compare them with the threshold. If it is less than the threshold, the arm is bent and there is a possibility of throwing or slashing; if it is greater than the threshold, the possibility of throwing or slashing is small.
8. The multi-target fusion dangerous behavior recognition algorithm according to claim 7 is characterized by: In step S3-2-4, a multi-frame prediction mechanism is adopted, and the formula is: in, Represents the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of n frames, α 11 ⊙α 21 This is the calculation formula for the unit vector inner product of the shoulder-elbow vector and the elbow-wrist vector of the first frame.
