Abnormal limb behavior detection method, device, terminal and medium
Through the example segmentation model, identify the key points of the human body and compare the center distance and reference distance, combine the swing amplitude to detect abnormal limb behavior, solving the problems of poor timeliness and poor reliability caused by relying on third-person reports in the existing technology, and achieving abnormalities Timely and reliable detection and alerting of physical behavior.
Patent Information
- Application Number
- CN202110349520.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-31
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-03-31
AI Technical Summary
In the prior art, abnormal physical behaviors such as fighting and fighting are usually only discovered in a timely manner by third-party reports, resulting in poor timeliness and poor reliability. Especially behaviors that occur in hidden places may lead to untimely discovery and pose a threat to social and personal security.
The example segmentation model is used to process the image, identify key points of the human body, and detect abnormal limb behavior through the comparison of the central distance and reference distance, and combine the swing amplitude and relative position information to determine whether there are suspected abnormal behaviors and issue an alarm.
Timely and reliable detection of abnormal physical behaviors is achieved, the efficiency and accuracy of discovery and processing are improved, and the security risks brought about by delayed reporting are reduced.
Smart Images

Figure CN113111747B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, terminal and medium for detecting abnormal limb behavior. Background Art
[0002] With the improvement of people's quality and the improvement of social security, the incidence of fighting is gradually decreasing, but fighting still poses a great risk to people's life and property safety. Usually, abnormal physical behaviors such as fighting can only be discovered in time with the report of a third party. For abnormal physical behaviors that occur in some hidden places, due to the lack of people, the discovery of the behavior may be untimely and unreliable, causing certain harm to society and others. Summary of the invention
[0003] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a method, device, terminal and medium for detecting abnormal limb behavior, which are used to solve the technical problem that abnormal limb behavior can only be discovered by relying on reports from third parties, which has poor timeliness and reliability.
[0004] In view of the above problems, the present invention provides a method for detecting abnormal limb behavior, comprising:
[0005] Acquire an image to be identified, and perform instance segmentation processing on the image to be identified through an instance segmentation model to distinguish various target objects in the image to be identified and identify key points of the human body of the target object;
[0006] Respectively obtain position information of human body key points of a first target object and position information of human body key points of a second target object, wherein the first target object is adjacent to the second target object, and the human body key points include a central key point and a reference key point;
[0007] Determining a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object;
[0008] Abnormal limb behavior is detected based on the size relationship between the center distance and the reference distance.
[0009] Optionally, the central key point includes a neck key point, the reference key point includes a left shoulder key point and a right shoulder key point, the reference distance includes a sum of a shoulder width of the first target object and a shoulder width of the second target object, the shoulder width includes a distance between the left shoulder key point and the right shoulder key point, and the detecting of abnormal limb behavior between the first target object and the second target object based on a size relationship between the central distance and the reference distance includes:
[0010] If the center distance is less than the reference distance, it is determined that there is suspected abnormal limb behavior.
[0011] Optionally, the human body key points further include swinging part key points, and the abnormal limb behavior detection method further includes:
[0012] Acquire a comparison image to be identified, and perform instance segmentation processing on the comparison image to be identified by using the instance segmentation model to distinguish target objects in the comparison image to be identified and identify key points of the human body of the target objects, wherein the comparison image to be identified is shot before or after the shooting time of the image to be identified, and the comparison identification image includes the first target object and / or the second target object;
[0013] The swing amplitude of the first target object and / or the second target object is obtained, and the swing amplitude is determined according to the position information of the key points of the swing parts of the first target object and / or the second target object, the position information of the center key point, and the position information of the reference key point.
[0014] Optionally, at least one of the following is also included:
[0015] The basic distance is determined according to the reference key points, the reference key points include the left shoulder key point and the right shoulder key point, and the basic distance is determined in the following manner:
[0016]
[0017] The relative position information is determined according to the position information of the key point of the swinging part and the position information of the central key point, wherein the relative position information includes first relative position information and second relative position information, and the first relative position information is determined in the following manner:
[0018] x1'=x 10a -x 17a , y1'=y 10a -y 17a ,
[0019] The second relative position information is determined by:
[0020] x0'=x10b -x 17b , y0'=y 10b -y 17b ;
[0021] The swing amplitude is determined by:
[0022]
[0023] Among them, ratio is the swing amplitude, L0 is the basic distance, x5 is the horizontal coordinate of the right shoulder key point, y5 is the vertical coordinate of the right shoulder key point, x6 is the horizontal coordinate of the left shoulder key point, y6 is the vertical coordinate of the left shoulder key point, x1' is the relative horizontal coordinate of the target object in the image to be identified, y1' is the relative vertical coordinate of the target object in the image to be identified, x0' is the relative horizontal coordinate of the target object in the contrast image to be identified, y0' is the relative vertical coordinate of the target object in the contrast image to be identified, x 10a is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x 17a is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10a is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17a is the ordinate of the central key point of the target object in the image to be identified, x 10b is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x 17b is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10b is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17b It is the ordinate of the central key point of the target object in the image to be identified.
[0024] Optionally, the swinging part key points include at least one of the left hand key points, the left elbow key points, the right hand key points, and the right elbow key points, and the abnormal limb behavior detection method further includes:
[0025] If the swinging part key points include at least two human body key points, obtaining an average swing amplitude as a reference swing amplitude;
[0026] If the swinging part key point includes a human body key point, the swing amplitude corresponding to the swinging part key point is used as a reference swing amplitude;
[0027] If the reference swing amplitude of at least one of the target objects is greater than a preset amplitude threshold, it is determined that there is suspected abnormal limb behavior.
[0028] Optionally, the instance segmentation model is obtained in the following way:
[0029] Obtain a human body image with human body key points annotated;
[0030] The key points of the human body are converted into the maximum activation points of the matrix through Gaussian distribution;
[0031] Inputting the human body image into the backbone network to generate an output matrix;
[0032] A loss function is determined according to the maximum activation point and the output matrix, and the backbone network is trained.
[0033] Optionally, the abnormal limb behavior detection method further includes:
[0034] If the detection results include suspected abnormal limb behavior, an alarm is issued.
[0035] The present invention also provides an abnormal limb behavior detection device, comprising:
[0036] An image acquisition module is used to acquire an image to be identified, and perform instance segmentation processing on the image to be identified through an instance segmentation model to distinguish each target object in the image to be identified and identify the key points of the human body of the target object;
[0037] A position acquisition module, used to respectively acquire position information of a human body key point of a first target object and position information of a human body key point of a second target object, wherein the first target object is adjacent to the second target object, and the human body key points include a central key point and a reference key point;
[0038] A determination module, configured to determine a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object;
[0039] The detection module is used to detect abnormal limb behavior according to the size relationship between the center distance and the reference distance.
[0040] The present invention also provides a terminal, comprising a processor, a memory and a communication bus;
[0041] The communication bus is used to connect the processor and the memory;
[0042] The processor is used to execute the computer program stored in the memory to implement the abnormal limb behavior detection method.
[0043] The present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to enable a computer to execute the abnormal limb behavior detection method.
[0044] As described above, the abnormal limb behavior detection method, device, terminal and medium provided by the present invention have the following beneficial effects:
[0045] The method obtains an image to be identified and performs instance segmentation processing on the image to be identified through an instance segmentation model to distinguish various target objects in the image to be identified and the human body key points of the identified target objects; obtains the position information of the human body key points of the first target object and the position information of the human body key points of the second target object respectively; determines the center distance between the center key point of the first target object and the center key point of the second target object, and the reference distance between the reference key point of the first target object and the reference key point of the second target object; detects abnormal limb behavior according to the size relationship between the center distance and the reference distance, thereby solving the technical problem that abnormal limb behavior can only be discovered by relying on reports from third parties, which is poor in timeliness and reliability. The method is convenient, timely, fast, reliable and more objective. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of a flow chart of an abnormal limb behavior detection method provided in Embodiment 1 of the present invention;
[0047] Figure 2 A schematic diagram of a human body key point determined according to the human skeleton joint points;
[0048] Figure 3 A schematic diagram of a target object and human key points of the target object identified after being processed by an instance segmentation model;
[0049] Figure 4 A specific flow chart of the abnormal limb behavior detection method provided in the first embodiment of the present invention;
[0050] Figure 5 A schematic diagram of the structure of an abnormal driving behavior detection device provided in Embodiment 2 of the present invention;
[0051] Figure 6 A schematic diagram of the structure of a terminal provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION
[0052] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0053] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.
[0054] Embodiment 1
[0055] See also Figure 1 , an abnormal limb behavior detection method provided by an embodiment of the present invention includes:
[0056] S101: Acquire an image to be identified, and perform instance segmentation processing on the image to be identified through an instance segmentation model to distinguish various target objects in the image to be identified and identify human key points of the target objects.
[0057] The image to be identified can be obtained through monitoring images or through capture devices, and can be a real-time image or a non-real-time image. When the image to be identified is a real-time image, the abnormal body behavior detection method is real-time monitoring, which helps to detect abnormal body behavior in time, issue an alarm, and take timely action to stop it.
[0058] Abnormal physical behavior includes but is not limited to fighting.
[0059] The image to be recognized can be a picture or a frame of a video.
[0060] The output result of the instance segmentation model can be to identify all the human body key points of the target object, or to identify some of the human body key points required later, such as only identifying the center key points, reference key points, swinging part key points, etc.
[0061] In some embodiments, the instance segmentation model may be obtained by:
[0062] Obtain a human body image with human body key points annotated;
[0063] The key points of the human body are converted into the maximum activation points of the matrix through Gaussian distribution;
[0064] Input the human image into the backbone network to generate the output matrix;
[0065] The loss function is determined based on the maximum activation point and the output matrix, and the backbone network is trained.
[0066] Optionally, the backbone network includes but is not limited to the network of mobilenetv2, the key points of the human body are converted into the maximum activation points in a 64*64 matrix through Gaussian distribution to obtain a 21*64*64 gt heatmap, the human image is input into the network of mobilenetv2 to obtain a 21*64*64 output matrix, L2loss is determined as the loss function according to the output matrix and the 21*64*64 gtheatmap, and the mobilenetv2 network is optimized by the loss function.
[0067] Optionally, the loss function can be determined based on the distance from the maximum activation point to the corresponding point mapped on the output result.
[0068] Optionally, the loss function includes but is not limited to L2loss.
[0069] The above instance segmentation model uses SOLO's instance segmentation method to decouple the original mask prediction into kernel learning and feature learning. It does not require anchor boxes, standardization, or bounding box detection. It directly maps the input image to be identified to the required object class and object mask, making training and reasoning much simpler. The key points of the human body are converted into the maximum activation points in the matrix through Gaussian distribution. The backbone network outputs the matrix, which is optimized through L2loss to be consistent with the label.
[0070] The backbone network outputs the human body mask and key points at the same time. This method is based on the principle of image segmentation. The reasoning speed will not slow down as the number of human bodies in the image increases. It has strong real-time performance and is suitable for a variety of scenarios, such as indoor places such as gyms, places with large traffic such as subway stations, etc.
[0071] Optionally, the target object includes a human body. The posture of the target object is not limited here, and can be a variety of postures such as standing, lying, sitting, etc.
[0072] The positions of the key points of the human body can be determined based on the positions of the human skeleton, see Figure 2 and Figure 3 , Figure 2 It is a schematic diagram of key points of the human body determined according to the joint points of the human skeleton. Figure 3 A schematic diagram of the target object and the human key points of the target object identified after being processed by the instance segmentation model.
[0073] It should be noted that Figure 2 The serial number is only an example, and those skilled in the art can change the identification method of the serial number as needed, including but not limited to using letters, words, etc. for identification.
[0074] Figure 2 The identification method of the human body key points in is only an example. Those skilled in the art can add and define other feature points as human body key points, or delete some human body key points as needed.
[0075] Optional, continue to see Figure 2 , the key points of the human body include but are not limited to the left eye key 2, the right eye key point 1, the left ear key point 4, the right ear key point 3, the mouth key point 0, the neck key point 17, the left shoulder key point 6, the right shoulder key point 5, the left elbow key point 8, the right elbow key point 7, the left hand key point 10, the right hand key point 9, the left hip key point 12, the right hip key point 11, the left knee key point 14, the right knee key point 13, the left foot key point 16, and the right foot key point 15.
[0076] S102: Acquire the position information of the human body key points of the first target object and the position information of the human body key points of the second target object respectively.
[0077] It should be noted that the first target object is adjacent to the second target object, and the key points of the human body include the central key points and the reference key points.
[0078] The first target object and the second target object are two adjacent target objects in the image to be recognized, and the two may be adjacent front to back, or adjacent left to right.
[0079] The position information of the key points of the human body of each target object can be represented by position data based on the same coordinate system.
[0080] The center key point and the reference key point can be human body key points specified by a person skilled in the art among various human body key points. Optionally, the center key point can be a human body key point with the smallest swing amplitude during the march among various human body key points, and the reference key point can be a human body key point that can indicate the left and right width of the human body.
[0081] The central key points of each target object are all key points of the same part, and the reference key points of each target object are all key points of the same part.
[0082] S103: Determine a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object.
[0083] The center distance may be determined according to the position information of the center key point of the first target object and the position information of the center key point of the second target object, and the center distance may indicate the current distance between the first target object and the second target object.
[0084] A reference distance may be determined based on the position information of the reference key points of the first target object and the second target object, and the reference distance may indicate a reasonable distance between the first target object and the second target object, or a safe distance between the first target object and the second target object.
[0085] S104: Detect abnormal limb behavior based on the size relationship between the center distance and the reference distance.
[0086] If the center distance is less than the reference distance, it means that the distance between the first target object and the second target object is too close, and there may be abnormal limb behavior, that is, there is suspected abnormal limb behavior.
[0087] If the center distance is greater than or equal to the reference distance, it means that the distance between the first target object and the second target object is appropriate, and there may be no abnormal limb behavior.
[0088] Since the reference distance is determined based on the reference key points of the first target object and the reference key points of the second target object, the reference distance essentially achieves the goal of "different for different people" instead of being measured by a certain unified data, making the detection of abnormal limb behavior more accurate.
[0089] In some embodiments, the central key point includes a neck key point, the reference key point includes a left shoulder key point and a right shoulder key point, the reference distance includes the sum of a shoulder width of the first target object and a shoulder width of the second target object, the shoulder width includes a distance between the left shoulder key point and the right shoulder key point, and according to the size relationship between the central distance and the reference distance, detecting abnormal limb behavior between the first target object and the second target object includes:
[0090] If the center distance is less than the reference distance, it is determined that there is suspected abnormal limb behavior.
[0091] Generally speaking, when a person is moving, the neck is in a relatively central position. By determining the distance between the necks of two adjacent target objects, the current distance relationship between the two can be effectively indicated. The width of the shoulder does not change much during movement. Therefore, the sum of the shoulder widths between two adjacent target objects is used as the reference distance to better represent the reasonable "safe distance" between the first target object and the second target object.
[0092] However, sometimes the environment is crowded, which may cause the center distance between the first target object and the second target object to be smaller than the reference distance, but there is no abnormal limb behavior between the two. Therefore, other parameters are needed to further detect whether there is abnormal limb behavior in order to improve the detection accuracy of abnormal limb behavior.
[0093] In some embodiments, the key points of the human body further include key points of the swinging parts, and the abnormal limb behavior detection method further includes:
[0094] Acquire a comparison image to be identified, and perform instance segmentation processing on the comparison image to be identified by using an instance segmentation model to distinguish target objects in the comparison image to be identified and identify key points of the human body of the target object, wherein the comparison image to be identified is shot before or after the shooting time of the image to be identified;
[0095] The swing amplitude of the first target object and / or the second target object is obtained, and the swing amplitude is determined according to the position information of the key points of the swing parts of the first target object and / or the second target object, the position information of the central key point, and the position information of the reference key point.
[0096] The swing amplitude of the first target object or the swing amplitude of the second target object may be acquired separately as required.
[0097] However, in some fighting situations, one party may be at a disadvantage and unable to resist, so the reliability of the method of obtaining the swing amplitude of a certain target object alone is relatively poor, and the reliability of the method of simultaneously obtaining the swing amplitude of the first target object and the swing amplitude of the second target object is poor.
[0098] In some embodiments, obtaining the swing amplitude of the first target object and / or the second target object includes:
[0099] Acquire relative position information of the first target object and / or the second target object, the relative position information including first relative position information of the key point of the swing part relative to the central key point in the image to be identified, and second relative position information of the key point of the swing part relative to the central key point in the image to be identified;
[0100] Obtaining a basic distance corresponding to the relative position information, where the basic distance is determined according to the position information of the reference key point;
[0101] The swing amplitude is determined according to the basic distance, the first relative position information and the second relative position information, and abnormal limb behavior is detected.
[0102] The compared image to be identified includes at least one of the first target object and the second target object.
[0103] The comparison image to be identified can be an image taken at a different time, the same position, and the same shooting angle as the image to be identified for the same batch of target objects. It should be noted that the image to be identified and the comparison object to be identified can be taken by the same shooting device or by different shooting devices.
[0104] The method of performing instance segmentation processing on the compared image to be identified through the instance segmentation model is similar to the method of performing instance segmentation processing on the image to be identified through the instance segmentation model. On the one hand, it can distinguish the various target objects in the compared image to be identified, and on the other hand, it can also identify the human body key points and their positions of each target object.
[0105] Optionally, the key points of the swinging parts may be key points corresponding to the parts that swing during the movement, including but not limited to hands, elbows, legs and feet.
[0106] Optionally, the relative position information includes first relative position information and second relative position information, and the relative position information can be determined by the position of the key point of the swinging part relative to the center key point.
[0107] The key point of the swing part in the image to be identified corresponds to the position of the key point of the swing part in the comparison image to be identified. For example, if the key point of the swing part is the left hand key point, then the first relative position information is the position of the left hand key point in the image to be identified relative to the center key point in the image to be identified, and the first relative position information is the position of the left hand key point in the comparison image to be identified relative to the center key point in the comparison image to be identified.
[0108] Optionally, the reference key points include key points of two symmetrically located human body parts, such as a left hand and a right hand, a left shoulder and a right shoulder, a left foot and a right foot, etc.
[0109] Optionally, only the relative position information of the first target object can be obtained, and then the basic distance of the first target object can be obtained to determine the swing amplitude and reference swing amplitude of the first target object. Alternatively, the relative position information of the second target object can be obtained, and then only the basic distance of the second target object can be obtained to determine the swing amplitude and reference swing amplitude of the second target object. Alternatively, both the relative position information of the first target object and the relative position information of the second target object can be obtained, and finally the reference swing amplitude of the first target object and the reference swing amplitude of the second target object can be determined respectively.
[0110] The basic distance corresponding to the relative position information, that is, the target object on which the relative position information is based, is determined by the reference key point of the target object on which the basic distance is based.
[0111] Optionally, the basic distance may be determined only based on the position information of the reference key points in the object to be identified, or may be determined only based on the position information of the reference key points in the image to be identified. Sometimes, in order to ensure the accuracy of detection, the first basic distance may be determined based on the position information of the reference key points in the object to be identified, and the second basic distance may be determined based on the position information of the reference key points in the image to be identified, and the average value between the first basic distance and the second basic distance may be obtained as the basic distance.
[0112] Optionally, the swing amplitude may be determined by first determining the movement amplitude according to the first relative position information and the second relative position information, and then determining the swing amplitude according to the movement amplitude and the basic distance.
[0113] Optionally, a basic distance is determined based on reference key points, the reference key points include a left shoulder key point and a right shoulder key point, and the basic distance is determined as follows:
[0114]
[0115] Among them, L0 is the basic distance, x5 is the horizontal coordinate of the right shoulder key point, y5 is the vertical coordinate of the right shoulder key point, x6 is the horizontal coordinate of the left shoulder key point, and y6 is the vertical coordinate of the left shoulder key point.
[0116] The left shoulder key points and the right shoulder key points can be in the image to be identified or in the image to be compared. This is not limited here. However, it should be noted that if the position information of the left shoulder key points is the coordinates in the image to be identified, the position information of the right shoulder key points should also use the coordinates in the image to be identified. Similarly, if the position information of the left shoulder key points is the coordinates in the image to be compared, the position information of the right shoulder key points should also use the coordinates in the image to be compared.
[0117] Optionally, relative position information is determined according to position information of key points of the swinging part and position information of the central key point, the relative position information includes first relative position information and second relative position information, and the first relative position information is determined in the following manner:
[0118] x1'=x 10a -x 17a Formula (2);
[0119] y1'=y 10a -y 17a Formula (3);
[0120] Among them, x1' is the relative horizontal coordinate of the target object in the image to be identified, y1' is the relative vertical coordinate of the target object in the image to be identified, and x 10a is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x17a is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10a is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17a is the ordinate of the central key point of the target object in the image to be identified.
[0121] Optionally, the second relative position information is determined in the following manner:
[0122] x0'=x 10b -x 17b Formula (4);
[0123] y0'=y 10b -y 17b Formula (5);
[0124] Among them, x0' is the relative horizontal coordinate of the target object in the image to be identified, y0' is the relative vertical coordinate of the target object in the image to be identified, and x 10b is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x 17b is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10b is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17b It is the ordinate of the central key point of the target object in the image to be identified.
[0125] Optionally, the swing amplitude is determined as follows:
[0126]
[0127] Among them, ratio is the swing amplitude, L0 is the basic distance, x1' is the relative horizontal coordinate of the target object in the image to be identified, y1' is the relative vertical coordinate of the target object in the image to be identified, x0' is the relative horizontal coordinate of the target object in the contrast image to be identified, and y0' is the relative vertical coordinate of the target object in the contrast image to be identified.
[0128] In some embodiments, the swinging part key point includes at least one of a left hand key point, a left elbow key point, a right hand key point, and a right elbow key point, and the abnormal limb behavior detection method further includes:
[0129] If the key points of the swinging part include at least two key points of the human body, obtaining an average swing amplitude as a reference swing amplitude;
[0130] If the key point of the swinging part includes a key point of the human body, the swing amplitude corresponding to the key point of the swinging part is used as the reference swing amplitude;
[0131] If the reference swing amplitude of at least one target object is greater than a preset amplitude threshold, it is determined that there is suspected abnormal limb behavior.
[0132] Optionally, the average swing amplitude may be an arithmetic average of the swing amplitudes corresponding to the key points of the human body, or may be a weighted average.
[0133] If the reference swing amplitude of one of the first target object and the second target object is greater than the preset amplitude threshold, it is determined that there is suspected abnormal limb behavior. If the center distance is less than the reference distance, it can be determined that there is abnormal limb behavior between the first target object and the second target object.
[0134] Optionally, the key points of the swinging part also include at least one of the left foot key point, the right foot key point, the left knee key point, the right knee key point, etc.
[0135] In some embodiments, the abnormal limb behavior detection method further includes:
[0136] The position information of the key points of the swinging part is obtained. If the position of the key points of the swinging part exceeds the preset safe position range, it is determined that there is suspected abnormal limb behavior.
[0137] The preset safe position range can be determined according to the current detection environment. For example, on the road, everyone is walking, and usually the height of the elbows and hands will not exceed the central key point such as the neck. Therefore, the position below the neck is the safe range of the elbows and hands. If the elbow or hand of a target object is detected to be higher than the central key point, it means that there is a high probability of abnormal limb behavior. For another example, the height of the feet is usually lower than the knees. Therefore, the position below the knees when standing normally is the safe range of the feet. If the height of the feet is detected to be higher than the knees, it means that there is a high probability of abnormal limb behavior.
[0138] In some embodiments, if suspected abnormal limb behavior is determined to exist from at least two dimensions (center distance is less than reference distance, reference swing amplitude is greater than preset swing amplitude, key points of swing part are beyond preset safe position range), then abnormal limb behavior is determined to exist.
[0139] In some embodiments, the abnormal limb behavior detection method further includes:
[0140] If the detection results include suspected abnormal limb behavior, an alarm is issued.
[0141] Optionally, the alarm also includes an alarm level, which can be determined based on the specific type and degree of suspected abnormal limb behavior, where the specific types of suspected abnormal limb behavior include a reference swing amplitude that is too large and a center distance that is less than a reference distance; the degree of suspected abnormal limb behavior includes a reference swing amplitude that is greater than a warning amplitude, where the warning amplitude is greater than a preset amplitude threshold, or the center distance is less than half of the reference distance, etc.
[0142] In some embodiments, the size relationship between the center distance and the reference distance and the swing amplitude may be determined in no particular order, that is, the swing amplitude may be determined first, and then the size relationship between the center distance and the reference distance may be determined.
[0143] In some embodiments, if the center distance is less than the reference distance, after determining that there is suspected abnormal limb behavior, the method further includes:
[0144] Acquire a comparison image to be identified, and perform instance segmentation processing on the comparison image to be identified by using an instance segmentation model to distinguish target objects in the comparison image to be identified and key points of the human body of the target object to be identified, wherein the comparison image to be identified is shot before or after the shooting time of the image to be identified, and the comparison image to be identified includes a first target object and a second target object;
[0145] Acquire relative position information, the relative position information including first relative position information of key points of the swinging part in the image to be identified, and compare second relative position information of key points of the swinging part in the image to be identified;
[0146] Determine the base distance based on the reference key points of the target object;
[0147] Determine the swing amplitude according to the basic distance, the first relative position information and the second relative position information, and detect abnormal limb behavior;
[0148] If the reference swing amplitude is greater than the preset amplitude threshold, it is determined that there is suspected abnormal limb behavior.
[0149] Optionally, whether abnormal limb behavior exists can be determined from at least one of two dimensions: the reference swing amplitude and the size relationship between the center distance and the reference distance.
[0150] If the existence of suspected abnormal limb behavior is determined from both dimensions, then the existence of abnormal limb behavior is determined. If the existence of suspected abnormal limb behavior is determined from only one dimension, it is also necessary to determine whether there is abnormal limb behavior in combination with the degree of the suspected abnormal limb behavior. For example, if the center distance is less than the reference distance, and the center distance is less than the preset percentage of the reference distance (50% of the reference distance value), then it can be determined that there is abnormal limb behavior. For another example, if the reference swing amplitude is greater than the preset amplitude threshold and greater than the warning amplitude, then it can also be determined that there is abnormal limb behavior.
[0151] In some embodiments, after determining the existence of suspected abnormal limb behavior, a comparison image to be identified including the first target object and the second target object can be further obtained, and then the swing amplitude of at least one of the first target object and the second target object can be determined respectively to further detect the abnormal limb behavior.
[0152] If the swing amplitude of at least one of the first target object and the second target object is greater than a preset amplitude threshold, it can be determined that abnormal limb behavior exists.
[0153] The abnormal limb behavior detection method mentioned in this embodiment is illustrated below by using a specific example.
[0154] S401: Acquire an image to be identified, and perform instance segmentation processing on the image to be identified using an instance segmentation model.
[0155] Continue to see Figure 3 ,by Figure 3 For example, the image to be identified includes two people, the first person (the person on the left of the image) and the second person (the person on the right of the image). The key points of each person's body can be seen Figure 3 shown.
[0156] S402: Obtain the key point position information of the left shoulder of the first person, the key point position information of the right shoulder of the first person, the key point position information of the left shoulder of the second person, and the key point position information of the right shoulder of the second person, and determine the reference distance.
[0157] At this time, the left shoulder key point and the right shoulder key point are used as reference key points.
[0158] Assume that the left shoulder key point position information of the first person is (x 5a ,y 5a ), the key point position information of the first person’s right shoulder is (x 6a ,y 6a ), the left shoulder key point position information of the second person is (x 5b ,y 5b ), the key point position information of the second person’s right shoulder is (x 6b ,y 6b), then the shoulder width L1 of the first person is:
[0159] L1=sqrt((x 5a -x 6a ) 2 +(y 5a -y 6a ) 2 ) formula (7);
[0160] The shoulder width L2 of the second person is:
[0161] L2=sqrt((x 5b -x 6b ) 2 +(y 5b -y 6b ) 2 ) formula (8);
[0162] Reference distance L c For: L c =L1+L2.
[0163] S403: Obtain the key point position information of the first person's neck and the key point position information of the second person's neck, and determine the center distance.
[0164] Assume that the key point position information of the first person's neck is (x 17a ,y 17a ), the key point position information of the second person’s neck is (x 17b ,y 17b ), then the center distance O is:
[0165] O = sqrt((x 17a -x 17b ) 2 +(y 17a -y 17b ) 2 ) formula (9);
[0166] S404: Determine whether the center distance is less than the reference distance, if so, execute step S405.
[0167] Optionally, if the center distance is greater than or equal to the reference distance, the process may be terminated, or step S405 may be executed.
[0168] S405: Determine the presence of suspected abnormal limb behavior.
[0169] S406: Obtain a comparison image to be identified, and perform instance segmentation processing on the comparison image to be identified using an instance segmentation model.
[0170] Optionally, the image to be identified includes a first person and a second person.
[0171] Optionally, the image to be identified is compared with an image frame extracted from a same video, and the image to be identified is compared with a plurality of image frames before the image to be identified.
[0172] Optionally, the method of performing instance segmentation processing on the image to be identified by using the instance segmentation model is similar to the method of performing instance segmentation processing on the image to be identified by using the instance segmentation model, and will not be repeated here.
[0173] S407: Obtain the relative position information of the first person and the relative position information of the second person respectively.
[0174] Taking the left hand key point as the swinging position key point as an example, the determination method of the relative position information is exemplified. Assume that the left hand key point of the first person in the image to be identified is (x 10c ,y 10c ), the key point of the neck of the first person in the image to be identified is (x 17a ,y 17a ), the left hand key point of the second person in the image to be identified is (x 10d ,y 10d ), the key point of the neck of the second person in the image to be identified is (x 17b ,y 17b ), the left hand key point of the first person in the image to be identified is (x 10e ,y 10e ), the key point of the neck of the first person in the image to be identified is (x 17e ,y 17e ), the left hand key point of the second person in the image to be identified is (x 10f ,y 10f ), the key point of the neck of the second person in the image to be identified is (x 17f ,y 17f ), then the first relative position information of the first person is (x2, y2), the first relative position information of the second person is (x3, y3), the second relative position information of the first person is (x4, y4), and the second relative position information of the second person is (x5, y5),
[0175] The determination methods of x2, x3, x4, x5, y2, y3, y4, and y5 are as follows:
[0176] x2=x 10c -x 17a Formula (10);
[0177] y2=y 10c -y 17a Formula (11);
[0178] x3=x 10d -x17b Formula (12);
[0179] y3=y 10d -y 17b Formula (13);
[0180] x4=x 10e -x 17e Formula (14);
[0181] y4=y 10e -y 17e Formula (15);
[0182] x5=x 10f -x 17f Formula (16);
[0183] y5=y 10f -y 17f Formula (17).
[0184] S408: Determine the swing amplitude.
[0185] Still taking the left hand key points of the first person and the second person as an example to illustrate the method of determining the swing amplitude, in step S402, the shoulder widths of the first person and the second person have been determined respectively, the shoulder width of the first person is L1, and the shoulder width of the second person is L2. At this time, the shoulder widths of the first person and the second person can be directly used as the basic distance.
[0186] The swing amplitude of the first person is:
[0187] ratio1 = sqrt((x2 - x4) 2 +(y2-y4) 2 ) / L1 formula (18).
[0188] The first person's swing amplitude is:
[0189] ratio2=sqrt((x3-x5) 2 +(y3-y5) 2 ) / L2 formula (19).
[0190] S409: Determine a reference swing amplitude.
[0191] If there is only one key point of the swing part, such as the key point of the left hand, the swing amplitude determined in step S409 is the reference swing amplitude. If there is more than one key point of the swing part, the swing amplitude of each key point of the swing part of each person is calculated respectively, and the average swing amplitude is obtained as the reference swing amplitude.
[0192] S410: Determine whether the reference swing amplitude is greater than a preset amplitude threshold, if so, execute step S411.
[0193] For example, the key points of the swinging part include the left hand key points and the right hand key points, so the swing amplitude of the left hand key points of the first person and the swing amplitude of the right hand key points of the first person are determined respectively, and the average swing amplitude of the two is obtained as the reference swing amplitude of the first person. Similarly, the swing amplitude of the left hand key points of the second person and the swing amplitude of the right hand key points of the second person are determined respectively, and the average swing amplitude of the two is obtained as the reference swing amplitude of the second person. If the reference swing amplitude of at least one of the first person and the second person is greater than the preset amplitude threshold, it is determined that abnormal limb behavior exists.
[0194] S411: Determine the presence of abnormal physical behavior.
[0195] Optionally, assuming that the preset amplitude threshold is 0.3, if the reference swing amplitude is greater than 0.3, it is judged that the swing repetition is too large; if the reference swing amplitude is less than 0.3, it is judged that the swing amplitude is normal.
[0196] Based on the instance segmentation method, each person in the image to be identified is segmented and their body key points are output. If the image to be identified contains two individuals, the distance between the two is too close, and the frequency of upper limb waving of both is high, the detection confirms the presence of abnormal limb behavior.
[0197] Optionally, an alarm will be sounded after abnormal physical behavior is detected.
[0198] In some embodiments, steps S406-S410 may be executed before, during, or after step S401. If the reference swing amplitude is greater than a preset amplitude threshold, steps S402-S404 are executed. If the center distance is determined to be less than the reference distance, abnormal limb behavior is determined to exist.
[0199] In some embodiments, after executing step S401, steps S402-S404 and steps S406-S410 may be executed simultaneously. If the reference swing amplitude is greater than the preset amplitude threshold, and the reference swing amplitude is greater than the preset amplitude threshold, it is determined that abnormal limb behavior exists.
[0200] The embodiment of the present invention provides an abnormal limb behavior detection method, which obtains an image to be identified and performs instance segmentation processing on the image to be identified through an instance segmentation model to distinguish each target object in the image to be identified and the human body key points of the identified target object; respectively obtains the position information of the human body key points of the first target object and the position information of the human body key points of the second target object; determines the center distance between the center key point of the first target object and the center key point of the second target object, and the reference distance between the reference key point of the first target object and the reference key point of the second target object; detects abnormal limb behavior according to the size relationship between the center distance and the reference distance, thereby solving the technical problem that abnormal limb behavior can only be discovered by relying on reports from third parties, which is poor in timeliness and reliability. The method is convenient, timely, fast, reliable, and more objective.
[0201] Embodiment 2
[0202] See also Figure 5 , an abnormal limb behavior detection device 1000, comprising:
[0203] The image acquisition module 1001 is used to acquire the image to be identified, and perform instance segmentation processing on the image to be identified through the instance segmentation model to distinguish each target object in the image to be identified and identify the key points of the human body of the target object;
[0204] A position acquisition module 1002, used to respectively acquire position information of a human body key point of a first target object and position information of a human body key point of a second target object, wherein the first target object is adjacent to the second target object, and the human body key point includes a central key point and a reference key point;
[0205] A determination module 1003 is used to determine a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object;
[0206] The detection module 1004 is used to detect abnormal limb behavior according to the size relationship between the center distance and the reference distance.
[0207] In this embodiment, the abnormal limb behavior detection device is essentially provided with multiple modules for executing the abnormal limb behavior detection method in the above embodiment. The specific functions and technical effects can be referred to the above embodiment 1, which will not be repeated here.
[0208] See also Figure 6 , an embodiment of the present invention further provides a terminal 1100, including a processor 1101, a memory 1102 and a communication bus 1103;
[0209] The communication bus 1103 is used to connect the processor 1101 and the memory 1102;
[0210] The processor 1101 is used to execute the computer program stored in the memory 1102 to implement the abnormal limb behavior detection method as described in any of the above embodiments.
[0211] The embodiment of the present invention further provides a computer-readable storage medium, characterized in that a computer program is stored thereon.
[0212] The computer program is used to enable a computer to execute the abnormal limb behavior detection method as described in any one of the above-mentioned embodiments.
[0213] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of the steps included in the embodiment 1 of the embodiment of the present application.
[0214] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0215] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0216] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0217] The flow chart and block diagram in the accompanying drawings illustrate the possible implementation architecture, function and operation of the method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0218] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.
Claims
1. A method for detecting abnormal limb behavior, characterized in that: include: Acquire an image to be identified, and perform instance segmentation processing on the image to be identified through an instance segmentation model to distinguish various target objects in the image to be identified and identify key points of the human body of the target object; Respectively obtain position information of human body key points of a first target object and position information of human body key points of a second target object, wherein the first target object is adjacent to the second target object, and the human body key points include a central key point and a reference key point; Determining a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object; Detecting abnormal limb behavior according to the size relationship between the center distance and the reference distance; The central key point includes a neck key point, the reference key point includes a left shoulder key point and a right shoulder key point, the reference distance includes a sum of a shoulder width of the first target object and a shoulder width of the second target object, the shoulder width includes a distance between the left shoulder key point and the right shoulder key point, and the detecting of abnormal limb behavior between the first target object and the second target object based on a size relationship between the central distance and the reference distance includes: If the center distance is less than the reference distance, it is determined that there is suspected abnormal limb behavior; Wherein, the human body key points also include swinging part key points, and the abnormal limb behavior detection method also includes: Acquire a comparison image to be identified, and perform instance segmentation processing on the comparison image to be identified by using the instance segmentation model to distinguish target objects in the comparison image to be identified and identify key points of the human body of the target objects, wherein the comparison image to be identified is shot before or after the shooting time of the image to be identified, and the comparison image to be identified includes the first target object and / or the second target object; Acquire a swing amplitude of the first target object and / or the second target object, wherein the swing amplitude is determined according to position information of key points of a swinging part of the first target object and / or the second target object, position information of a central key point, and position information of a reference key point; Wherein, the basic distance is determined according to the reference key points, the reference key points include the left shoulder key point and the right shoulder key point, and the basic distance is determined by the following method: The relative position information is determined according to the position information of the key point of the swinging part and the position information of the central key point, wherein the relative position information includes first relative position information and second relative position information, and the first relative position information is determined in the following manner: x1'=x 10a -x 17a, y1'=y 10a -and 17a , The second relative position information is determined by: x0'=x 10b -x 17b ,y0'=y 10b -and 17b ; The swing amplitude is determined by: Among them, ratio is the swing amplitude, L0 is the basic distance, x5 is the horizontal coordinate of the right shoulder key point, y5 is the vertical coordinate of the right shoulder key point, x6 is the horizontal coordinate of the left shoulder key point, y6 is the vertical coordinate of the left shoulder key point, x1' is the relative horizontal coordinate of the target object in the image to be identified, y1' is the relative vertical coordinate of the target object in the image to be identified, x0' is the relative horizontal coordinate of the target object in the contrast image to be identified, y0' is the relative vertical coordinate of the target object in the contrast image to be identified, x 10a is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x 17a is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10a is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17a is the ordinate of the central key point of the target object in the image to be identified, x 10b is the horizontal coordinate of the key point of the swing part of the target object in the image to be identified, x 17b is the horizontal coordinate of the central key point of the target object in the image to be identified, y 10b is the ordinate of the key point of the swing part of the target object in the image to be identified, y 17b It is the ordinate of the central key point of the target object in the image to be identified.
2. The abnormal limb behavior detection method according to claim 1, characterized in that: The swinging part key points include at least one of the left hand key points, the left elbow key points, the right hand key points, and the right elbow key points. The abnormal limb behavior detection method also includes: If the swinging part key points include at least two human body key points, obtaining an average swing amplitude as a reference swing amplitude; If the swinging part key point includes a human body key point, the swing amplitude corresponding to the swinging part key point is used as a reference swing amplitude; If the reference swing amplitude of at least one of the target objects is greater than a preset amplitude threshold, it is determined that there is suspected abnormal limb behavior.
3. The abnormal limb behavior detection method according to any one of claims 1 to 2, characterized in that: The instance segmentation model is obtained in the following way: Obtain a human body image with human body key points annotated; The key points of the human body are converted into the maximum activation points of the matrix through Gaussian distribution; Inputting the human body image into the backbone network to generate an output matrix; A loss function is determined according to the maximum activation point and the output matrix, and the backbone network is trained.
4. The abnormal limb behavior detection method according to any one of claims 1 to 2, characterized in that: The abnormal limb behavior detection method further comprises: If the detection results include suspected abnormal limb behavior, an alarm is issued.
5. An abnormal limb behavior detection device, characterized in that: include: An image acquisition module is used to acquire an image to be identified, and perform instance segmentation processing on the image to be identified through an instance segmentation model to distinguish each target object in the image to be identified and identify the key points of the human body of the target object; A position acquisition module, used to respectively acquire position information of a human body key point of a first target object and position information of a human body key point of a second target object, wherein the first target object is adjacent to the second target object, and the human body key points include a central key point and a reference key point; A determination module, configured to determine a center distance between a center key point of the first target object and a center key point of the second target object, and a reference distance between a reference key point of the first target object and a reference key point of the second target object; A detection module, used for detecting abnormal limb behavior according to the size relationship between the center distance and the reference distance; The central key point includes a neck key point, the reference key point includes a left shoulder key point and a right shoulder key point, the reference distance includes the sum of a shoulder width of the first target object and a shoulder width of the second target object, the shoulder width includes a distance between the left shoulder key point and the right shoulder key point, and the detection module is specifically used for: If the center distance is less than the reference distance, it is determined that there is suspected abnormal limb behavior.
6. A terminal, characterized in that: including a processor, a memory and a communication bus; The communication bus is used to connect the processor and the memory; The processor is used to execute the computer program stored in the memory to implement the abnormal limb behavior detection method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, The computer program is used to enable the computer to execute the abnormal limb behavior detection method according to any one of claims 1 to 4.
