Behavior analysis method and device, electronic equipment and computer program product

Abstract

The invention discloses a behavior analysis method and device, electronic equipment and a computer program product, and relates to a big data technology. The method comprises the following steps: processing received real-time video stream data to obtain continuous frame images; when it is detected that the continuous frames of images contain the target object, respectively extracting limb features of the target object in each frame of image based on a key point detection model; determining track features of the target object based on the limb features in each frame of image and the corresponding timestamp; and performing behavior analysis on the target object according to the limb features and the trajectory features. According to the method, the static limb state and the dynamic movement track of the target object are comprehensively considered, and the behaviors of the target object are accurately analyzed from multiple dimensions. Compared with the existing scheme of monitoring depending on manpower, the method has the advantages that the investment of labor cost is reduced, and the situations of false alarm and missing alarm are effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a behavior analysis method, device, electronic device and computer program product. Background Art

[0002] In today's society, as an important place for financial transactions, the security of banks is of vital importance. The frequent flow of personnel in banks and the diverse business operations pose various potential security risks, such as conflicts between personnel, malicious destruction of equipment, and tailing. These abnormal behaviors may not only cause serious losses to the property security of the bank, but also threaten the lives of customers and employees.

[0003] Traditional bank security monitoring mainly relies on manual review of the camera images, which has many limitations. On the one hand, manual monitoring requires a lot of manpower and time, and monitoring personnel are prone to fatigue and find it difficult to concentrate for a long time, which may miss some important behaviors. On the other hand, for complex scenes and behavior patterns, manual judgment often lacks accuracy and timeliness. Therefore, how to more intelligently monitor the behavior of the target objects within the monitoring range is a key issue that needs to be solved urgently. Summary of the invention

[0004] The present application provides a behavior analysis method, device, electronic device and computer program product, which can accurately detect the behavior of a target object under monitoring.

[0005] In a first aspect, the present application provides a behavior analysis method, comprising:

[0006] Process the received real-time video stream data to obtain continuous frame images;

[0007] When it is detected that the continuous frame images contain a target object, extracting limb features of the target object in each frame image based on a key point detection model;

[0008] Determine the trajectory features of the target object based on the limb features and the corresponding timestamps in each frame of the image;

[0009] Performing behavior analysis on the target object according to the limb features and the trajectory features.

[0010] In a second aspect, the present application provides a behavior analysis device, the device comprising:

[0011] The video data processing module is used to process the received real-time video stream data to obtain continuous frame images;

[0012] A limb feature extraction module, configured to extract limb features of the target object in each frame of image based on a key point detection model when it is detected that the consecutive frame images contain the target object;

[0013] A trajectory feature determination module, configured to determine trajectory features of the target object based on the limb features and corresponding timestamps in each frame of image;

[0014] A behavior analysis module, configured to perform behavior analysis on the target object according to the limb features and the trajectory features.

[0015] In a third aspect, the present application further provides an electronic device, which includes:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the behavior analysis method according to any embodiment of the present application.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by a processor, the behavior analysis method according to any embodiment of the present application is implemented.

[0020] In a fifth aspect, the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the behavior analysis method according to any embodiment of the present application is implemented.

[0021] The behavior analysis solution provided by the embodiments of the present application extracts limb features of the target object in each frame of image based on a key point detection model, details and obtains information such as the posture and actions of the target object. At the same time, combined with the trajectory features determined based on the limb features and timestamps, it comprehensively considers the static limb state and dynamic movement trajectory of the target object, and accurately analyzes the behavior of the target object from multiple dimensions. Compared with the existing solution that relies on manual monitoring, it reduces the input of labor costs and effectively reduces the situations of false alarms and missed alarms.

[0022] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium. Among them, the computer-readable storage medium may be packaged together with the processor of the behavior analysis device or packaged separately from the processor of the behavior analysis device, and the present application does not make any limitation thereto.

[0023] For the descriptions of the second, third, fourth, and fifth aspects in this application, reference may be made to the detailed description of the first aspect; moreover, for the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects, reference may be made to the analysis of the beneficial effects of the first aspect, and thus will not be elaborated here.

[0024] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of this application, nor is it used to limit the scope of this application. Other features of this application will become easily understandable through the following description.

[0025] It can be understood that before using the technical solutions disclosed in the embodiments of this application, the types, usage scopes, usage scenarios, etc. of the personal information involved in this application should be informed to the users and the authorization of the users should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of this application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can also be obtained based on these drawings without creative efforts.

[0027] Figure 1 is a schematic flowchart of a behavior analysis method provided by an embodiment of this application;

[0028] Figure 2 is another schematic flowchart of a behavior analysis method provided by an embodiment of this application;

[0029] Figure 3 is a schematic structural diagram of a behavior analysis device provided by an embodiment of this application;

[0030] Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to enable those skilled in the art to better understand the solutions of this application, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this embodiment. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification, claims and the above-mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0033] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described here are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that for the sake of description, only parts related to the present application rather than all structures are shown in the drawings.

[0034] Figure 1 It is a schematic flowchart of a behavior analysis method provided by an embodiment of the present application. This embodiment is applicable to real-time analysis of monitored video data to determine whether there is an abnormal behavior at the current moment. This method can be executed by a behavior analysis device, which can be implemented in the form of hardware and / or software and integrated in an electronic device that executes this method. Preferably, the electronic device in the embodiment of the present application can be a server or a computer device, etc.

[0035] Refer to Figure 1 , the behavior analysis method provided by this embodiment includes but is not limited to the following steps:

[0036] S110. Process the received real-time video stream data to obtain continuous frame images.

[0037] Real-time video stream data refers to digital signal data that is transmitted in real time from a video acquisition device (such as a camera) and exists in the form of a continuous sequence of video frames. These data contain image information that changes continuously over time and is usually transmitted at a certain frame rate (such as 25 frames per second or 30 frames per second, etc.). Image processing is to perform a series of processes on the received video stream data to extract relevant information that meets the application scenario.

[0038] Generally, video acquisition devices are deployed in each key area of a specific location to continuously collect video stream data; on the data processing server, video processing software is used to decode the received video stream, and continuous image frames are extracted at a fixed frame rate (such as 25 frames per second or 30 frames per second). The extracted image frames will serve as the basic data for subsequent detection and feature extraction; further, preprocessing operations can be performed on the extracted image frames, such as image scaling (adjusting the image to the input size required by the detection model), image normalization (mapping the pixel value range to between [0, 1] or [-1, 1] to improve the training and inference efficiency of the model), and image enhancement (such as brightness adjustment, contrast enhancement, etc.) to improve the image quality and facilitate the detection model to better identify target objects, etc. The specific processing operations are not limited here.

[0039] S120. When it is detected that the consecutive frame images contain a target object, limb features of the target object in each frame image are respectively extracted based on the key point detection model.

[0040] The target object refers to a specific object or person that needs to be concerned and analyzed in the video image. In this scenario, it mainly refers to a human object with limb movements and whose generated behaviors need to be analyzed.

[0041] The key point detection model is a model pre-trained based on deep learning in this embodiment, which is used to identify and locate the key parts or feature points of the target object in the image, such as the joint points of the human body (such as wrists, ankles, knees, elbows, etc.). The purpose of determining the key points is to describe the limb state and posture of the target object.

[0042] Specifically, when training the key point detection model, a large amount of image data in the bank scenario can be collected, including images of normal behaviors and various abnormal behaviors (such as armed threats, personnel conflicts, malicious damage to equipment, abnormal tailing, etc.), and the image data of various limb movements and postures of the target object in the images are used as the training data set, and the key points of the target object in the images are manually annotated, and the accurate position coordinates and other information of each key point are marked; the selected key point detection model is trained using the annotated data set. By adjusting the parameters of the model, such as the weights and biases of the neural network, etc., the model can learn the mapping relationship between the image features and the key point positions, so that when the model reaches the convergence condition, the key point detection model training is completed to achieve the purpose of accurately detecting the key points through the model.

[0043] Further, the method for separately extracting the limb features of the target object in each frame of image based on the key point detection model may be as follows: Input the image region containing the target object into the key point detection model, and the model will output the coordinate position information of each key point of the target object in the image. For example, for a human target, the model may output the coordinates of joints such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles; According to the detected key point coordinates, further calculate the limb features. For example, calculate the distance between two key points to obtain the limb length; Obtain the angle of the limb by calculating the included angle of the line connecting the key points; Analyze the change of the key point coordinates in adjacent frames to obtain the movement speed and direction of the limb, etc. More complex limb features can also be further extracted through vector operations, geometric transformations, etc. to comprehensively describe the limb movements and postures of the target object; Fuse the calculated various limb features to form a feature vector or feature matrix as the limb feature representation of the target object in this frame of image. This feature representation can be used for subsequent analysis of the behavior of the target object, action recognition, etc.

[0044] In this embodiment, the purpose of separately extracting the limb features of the target object in each frame of image through the key point detection model is that this embodiment is for detecting abnormal behaviors that occur in a real-time video stream. In some abnormal behaviors, such as armed threats, personnel conflicts, malicious damage to equipment, etc., corresponding limb movements are required. For example, it is necessary to determine whether the arm is in front of, above, or to the side of the body, and whether the legs are crossed. This embodiment analyzes these limb features to determine the overall posture of the target object (such as a defensive posture, an attacking posture, etc.), so as to achieve the purpose of analyzing whether there is an abnormal behavior.

[0045] S130. Determine the trajectory features of the target object based on the limb features in each frame of image and the corresponding timestamps.

[0046] For the obtained consecutive frames of images, each frame of image is marked with a corresponding timestamp. In this embodiment, the way of associating the timestamp with the limb features is to analyze the changes of the limb features between consecutive frames through the time features. That is, in this embodiment, the limb features provide behavior details, and the time features reflect the behavior trend: the limb features can show the behavior state of the target object at each timestamp in detail. For example, the angle at which the arm is bent in a certain frame may indicate that a grasping action is being performed. The time features, from the perspective of consecutive frames, determine the development trend of the action behavior generated by the target object by analyzing the changes of the limb features over time, so as to judge whether this grasping action is accidentally generated or an upcoming malicious behavior.

[0047] The trajectory feature is used to represent the moving direction and moving distance of the target object over time. In addition to the moving direction and moving distance, the trajectory feature in this embodiment may also include information reflecting the motion state of the target object, such as the moving speed and acceleration of the target object within a certain period of time. The purpose of analyzing information such as speed and acceleration is, for example, when it is detected that a target person in a video is running fast at a speed much higher than the normal walking speed, such as exceeding 3 meters per second, it may indicate an emergency or abnormal behavior, such as running away after a malicious snatch; another example is that the moving direction of the target person suddenly changes significantly and is not in a normal path planning or interaction scenario, such as a person who was originally queuing suddenly turns around and rushes towards the door or counter, which may mean that abnormal behavior is about to occur; another example is that within a normal activity area, the target person remains stationary for a long time, such as in a bank lobby, a customer stands or lies still for more than 15 minutes without queuing or handling business, which may indicate physical discomfort or other abnormal intentions, etc.

[0048] S140. Perform behavior analysis on the target object according to the limb feature and the trajectory feature.

[0049] In this embodiment, the limb feature and the trajectory feature obtained in real time are input into a behavior analysis model for behavior prediction. The result output by the behavior analysis model can be the probability that the current behavior belongs to an abnormal behavior; thus, according to the current output probability value, it is determined whether the behavior information of the target object is an abnormal behavior or not; optionally, the result output by the behavior analysis model can also be 0 or 1. 0 is used to indicate that through behavior analysis of the target object, it is shown that the target object does not have an abnormal behavior at present; 1 is used to indicate that through behavior analysis of the target object, it is shown that the target object has an abnormal behavior at present.

[0050] Optionally, feature comparison can also be performed through the behavior analysis model. For example, the limb feature and the trajectory feature of the target object extracted in real time are compared with a normal behavior model to calculate the similarity or difference degree between the current feature and the normal behavior feature; according to the feature comparison result and the set threshold, it is comprehensively judged whether the target object has an abnormal behavior. If the difference between the current feature and the normal behavior feature exceeds the set threshold, it is determined that the target object has an abnormal behavior; otherwise, it is determined that the target object does not have an abnormal behavior.

[0051] Furthermore, if it is determined that there is an abnormal behavior, the current abnormal behavior can be classified based on a behavior classification model to determine what type of abnormality it belongs to, such as armed threat, personnel conflict, malicious equipment damage, abnormal tailing, etc. Furthermore, corresponding linkage alarm mechanisms can be activated according to different types of abnormal behaviors, so as to improve the efficiency of handling abnormal behaviors.

[0052] The behavior analysis method provided by the embodiments of the present application can extract the limb features of the target object in each frame of the image based on the key point detection model, and can obtain detailed information such as the posture and actions of the target object. At the same time, combined with the trajectory features determined based on the limb features and timestamps, it comprehensively considers the static limb state and dynamic movement trajectory of the target object, analyzes the behavior of the target object from multiple dimensions, and can accurately analyze the behavior of the monitored target object. Compared with the existing solution that relies on manual monitoring, it reduces the input of labor costs and effectively reduces the situations of false alarms and missed alarms.

[0053] Figure 2 FIG. 4 is another schematic flowchart of the behavior analysis method provided by the embodiments of the present application. The embodiments of the present application are optimized based on the above embodiments. Specifically, the optimization is as follows: In this embodiment, the implementation processes of "extracting the limb features of the target object in each frame of the image based on the key point detection model", "determining the trajectory features of the target object based on the limb features and corresponding timestamps in each frame of the image", and "performing behavior analysis on the target object according to the limb features and trajectory features" in the above embodiments are explained in detail.

[0054] See Figure 2 FIG. 4, the behavior analysis method provided by this embodiment includes but is not limited to the following steps:

[0055] S210. Process the received real-time video stream data to obtain consecutive frame images.

[0056] S220. Obtain the contour information of the target object in each frame of the image.

[0057] The purpose of obtaining the contour information of the target object in each frame of the image is to help analyze the changes of each key bone point in the consecutive frame images in the subsequent steps. Specifically, the method of obtaining the contour information of the target object in each frame of the image can be: performing image segmentation based on the image processing method to obtain foreground information and background information, and then obtaining the contour information of the target object based on the foreground information; optionally, model training can also be performed based on the deep learning method, inputting each frame of the image into the trained model, the model outputs the segmentation mask of the target object (a binary image, and the pixel value indicates whether the pixel belongs to the target object), and then using the contour extraction method to extract the contour information, etc. The specific method of obtaining the contour information is not limited here. In this embodiment, by obtaining the contour information of the target object to define the range of the target object in the image and exclude background interference, the subsequent key point detection and limb feature analysis of the target object are more targeted and accurate, improving the accuracy and reliability of the entire system's behavior analysis of the target object.

[0058] In this embodiment, to reduce the data processing pressure, after obtaining consecutive frame images, the solution provided in this embodiment first triggers the operation of obtaining the contour information of the target object in each frame image when it is detected that the consecutive frame images generated within a preset time period all contain the same target object; if the consecutive frame images generated within the preset time period do not all contain the same target object, it indicates that the current object appears accidentally and no corresponding abnormal behavior will occur, so there is no need to execute the solution provided in this embodiment to obtain the next video stream for analysis, etc. Among them, the above-mentioned preset time period can be 5 seconds, 10 seconds, 30 seconds, etc., and the selection of the specific preset time period is not limited here.

[0059] S221. Based on the key point detection model, perform key point detection on the contour information in each frame image to obtain the key skeleton points and corresponding coordinate information of the target object in each frame image.

[0060] Input the contour area of the target object in each frame image into the trained key point detection model. The model analyzes the input image features and outputs the coordinate information of the key skeleton points of the target object. For example, for a human target, it may output the coordinates of joint points such as the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. In this embodiment, the key skeleton points and coordinate information are obtained through the key point detection model, which can accurately capture the limb structure and posture information of the target object and provide an accurate data basis for determining limb features. These key point information can be used to analyze the actions and posture changes of the target object and play an important role in identifying abnormal behaviors (such as attacking postures, falling postures, etc.).

[0061] In this embodiment, the key skeleton points include at least one.

[0062] S222. Determine the limb features of the target object according to the coordinate information corresponding to each key skeleton point in each frame image.

[0063] The current limb features can be, within consecutive frame images, the bending condition of the knees (such as excessive bending may indicate a wrestling phenomenon), the changes of the wrists and elbows (such as, if holding an object to smash, it will change continuously over time), etc. The specific way to obtain the corresponding limb features can be obtained through coordinate calculation.

[0064] The method for limb features provided in this embodiment performs key point detection based on contour information, and then obtains key skeletal points and coordinate information, which can accurately capture the limb structure of the target object; the key skeletal points serve as the identification of the key parts of the limb, and their accurate coordinate information provides a reliable data basis for the subsequent analysis of limb features; furthermore, the limb features are determined according to the coordinate information of the key skeletal points, which can comprehensively describe the limb state of the target object; by accurately determining the limb state of the target object, the subtle changes in the behavior of the target object can be captured more sensitively, the signs of abnormal behavior can be detected in time, and the accuracy and reliability of abnormal behavior detection can be improved.

[0065] A preferred embodiment, in this embodiment, the above step S222 can be implemented in the following manner:

[0066] For each frame of image, calculate the skeletal length information and joint angle information between adjacent skeletal points according to the coordinate information of the key skeletal points; extract the action features according to the skeletal distance information and joint angle information corresponding to consecutive frames of images; determine the limb features of the target object according to the action features. By this method, the limb state of the target object at each moment can be accurately described, which helps to analyze the behavior of the target object in detail; by analyzing the action features and limb features, the situations that do not conform to the normal behavior pattern can be found in time, which helps to identify abnormal behavior in time.

[0067] Specifically, in this embodiment, the method for calculating the skeletal length information can be: calculate the coordinate distance between the key skeletal points of the shoulder and the elbow to obtain the length of the upper arm. By comparing the changes in the skeletal length in different frames, the stretching or contracting state of the limb can be judged; the method for calculating the joint angle information can be: using the principle of trigonometric functions, calculate the joint angle according to the coordinate of the key skeletal points of adjacent bones. For example, calculate the angle of the elbow joint according to the coordinate of the key skeletal points of the shoulder, elbow and wrist. The change in the joint angle can intuitively reflect the change in the limb posture, such as the arm bending or straightening; further, the method for extracting the action features can be: analyze the changes in the limb features in consecutive frames of images, and extract the action features, such as calculating the change rate of the joint angle between adjacent frames to judge the limb movement speed and acceleration; analyze the change trend of the skeletal length to judge whether the limb is stretching or contracting. These action features can be used to identify specific action patterns, such as waving and kicking; furthermore, the method for determining the limb features of the target object according to the action features can be: when the elbow joint angle is close to 180 degrees, judge that the arm is in a straight state; when the angle is close to 90 degrees, judge that the arm is in a bent state. By comprehensively analyzing multiple joint angles and skeletal lengths, more complex limb postures, such as standing, sitting, running, etc., can be judged.

[0068] S230. Determine the center point coordinates of the target object based on the limb features in each frame of image.

[0069] Since the limb features of this embodiment include the coordinate information of each key bone point of the target object, the center of gravity of the target object can be determined by weighted averaging, and used as the center point coordinates. For a human target, the weights of different parts can be set according to their mass distribution. For example, the weight of the torso part is relatively large, and the weights of the limbs are relatively small. Optionally, when the target object is a human, the key point coordinates of the hip joint can be selected as the center point coordinates. The hip joint is located at the center of the human body and can better represent the overall position of the human body. The specific method for determining the center point coordinates of the target object is not limited here.

[0070] S231. Associate the center point coordinates in each frame of image with the corresponding time stamp to obtain the trajectory points in the consecutive frames of images.

[0071] During the video acquisition process, record the acquisition time of each frame of image as the time stamp. If the video frame rate is fixed, the time stamp can be calculated according to the frame number and the frame rate. When associating the center point coordinates and the time stamp, it is necessary to ensure that the center point coordinates of each frame of image accurately correspond to the time stamp of that frame to avoid data misalignment.

[0072] Furthermore, corresponding changes are made according to the change of the center point coordinates in each frame of image with the time stamp, forming the trajectory points of the target object in the consecutive frames of images.

[0073] S232. Determine the trajectory features according to the trajectory points in the consecutive frames of images.

[0074] The trajectory features in this embodiment may include the movement speed, movement direction, movement acceleration, trajectory curvature corresponding to the trajectory, etc. of the target object. By analyzing the current physical quantity information, it can effectively reflect how fast the target object moves in the monitoring area, the moving target point, and analyze whether the moving path is the driving path of regular customers, etc. For example, it can also analyze whether there are sudden acceleration, deceleration or turning actions.

[0075] In this embodiment, by determining the center point coordinates of the target object, the position of the target object in each frame of image can be accurately located, so as to achieve more accurate target tracking in the consecutive frames of images. Further, after associating the center point coordinates with the time stamp to form trajectory points, the movement process of the target object over time can be obtained in detail. By analyzing the trajectory features such as the movement path, speed and acceleration reflected by the trajectory points, the subtle changes in the movement of the target object can be captured, which helps to accurately assist in the analysis of abnormal behaviors.

[0076] In another preferred embodiment, in this embodiment, the above step S232 can be specifically implemented in the following manner:

[0077] In adjacent consecutive frame images, determine the moving displacement and motion direction of the target object according to the coordinate information of two adjacent trajectory points; determine the object speed of the target object between two adjacent trajectory points according to the moving displacement and time interval; determine the trajectory curvature according to three adjacent trajectory points; obtain trajectory features according to the moving displacement, motion direction, object speed, and trajectory curvature. In this embodiment, by calculating the object displacement, motion direction, object speed, and trajectory curvature, the motion state of the target object can be comprehensively and accurately described, providing rich and accurate information for in-depth analysis of the behavior of the target object.

[0078] Specifically, the method for determining the moving displacement of the target object according to the coordinate information of two adjacent trajectory points can be to obtain the coordinates of the two trajectory points and calculate the moving displacement by calculating the coordinates of the two trajectory points according to the distance formula between two points; furthermore, the motion direction can be obtained by calculating the angle between the vector between the two trajectory points and the horizontal direction. Further, on the basis of knowing the moving displacement in consecutive frame images, obtain the time interval between two adjacent frames of images, and based on the velocity formula, the object speed of the target object between two adjacent trajectory points can be obtained to obtain the information about the motion speed of the target object; finally, by obtaining the coordinate information of three adjacent trajectory points, the curve formed by fitting these three points can be obtained by the least squares method, and then according to the mathematical expression of the curve, sudden large displacements, rapid speed changes, abnormal changes in the motion direction, or abnormal fluctuations in the trajectory curvature may all imply the occurrence of abnormal situations. In this embodiment, by monitoring and analyzing these trajectory features, sudden large displacements, rapid speed changes, abnormal changes in the motion direction, or abnormal fluctuations in the trajectory curvature can be detected in a timely manner. By monitoring and analyzing these features, potential safety threats can be detected in a timely manner and early warnings can be issued.

[0079] S240. Determine the first similarity value corresponding to the standard limb feature according to the limb weight values corresponding to different limb parts in the limb feature.

[0080] In this embodiment, different limb weight values are assigned to different limb parts included in the limb feature, such as the head, arms, legs, etc. These weight values can be determined according to the actual application scenario and importance. For example, when judging abnormal walking, the weight of the legs may be relatively high; when judging the presence of smashing behavior, the arms have a higher weight; the specific weight assignment method is not limited here.

[0081] The specific method for determining the first similarity value corresponding to different limb parts and the standard limb feature can be calculated by methods such as Euclidean distance and cosine similarity. The specific method for calculating the similarity value is not limited here.

[0082] Further, multiply the similarity of each limb part by its corresponding limb weight value, and then sum them up to obtain the first similarity value.

[0083] Among them, the above standard limb features are features obtained through model training in advance. For example, in a self-service deposit-withdrawal machine, when a user has a withdrawal behavior, they usually stand normally and perform hand operations according to the device voice prompts, and leave after the withdrawal is completed; while when there are features such as slapping the device significantly and for a long time, or hitting and smashing with a stick in hand, the first similarity value is relatively small or approaches 0 when calculating the similarity with the standard limb features.

[0084] S241. Determine the corresponding second similarity value according to the trajectory feature and the standard trajectory feature.

[0085] In this embodiment, the method for determining the second similarity value corresponding to the trajectory feature and the standard trajectory feature can be implemented based on methods such as dynamic time warping and Mahalanobis distance. Taking the dynamic time warping method as an example, it can handle the situation where the trajectory lengths are different, calculate the similarity by finding the optimal matching path between two trajectories, and then obtain the second similarity value.

[0086] Among them, the standard trajectory feature is obtained through model training in advance. For example, in the case of regular bank business handling, the trajectory feature of a user is mostly entering the door - taking a number - waiting - counter - leaving the door; if it is detected that the current trajectory feature is an area where regular users are prohibited from entering, the second similarity value is relatively small or approaches 0 when calculating with the standard trajectory feature.

[0087] S242. Obtain the first weight corresponding to the limb feature and the second weight corresponding to the trajectory feature.

[0088] In this embodiment, different weight values are assigned to the limb feature and the trajectory feature, and the sum of the first weight and the second weight is 1. According to the actual application scenario and requirements, determine the relative importance of the limb feature and the trajectory feature in judging abnormal behaviors. For example, in some scenarios, abnormal limb movements may better reflect abnormal behaviors, and at this time, the weight of the limb feature can be set higher; while in other scenarios, abnormal changes in the trajectory may be more critical, and the weight of the trajectory feature can be increased accordingly; the specific distribution method of the first weight and the second weight is not limited here.

[0089] S243. Obtain the abnormal value of the target object according to the first weight, the first similarity value, the second weight, and the second similarity value.

[0090] According to the method of weighted summation, comprehensively calculate the abnormal value of the target object.

[0091] S244. When the abnormal value exceeds the preset threshold, determine that the behavior analysis result of the target object is abnormal behavior.

[0092] The preset threshold in this embodiment is obtained through a large number of experiments and data analyses, and this threshold can be adjusted according to different application scenarios and acceptable false alarm rates and missed alarm rates. For example, if the preset threshold is 0.5, when the currently calculated abnormal value exceeds the preset threshold, it can be determined that the target object has abnormal behavior.

[0093] The method for determining abnormal behavior provided in this embodiment, by using the method of fusing limb features and trajectory features to judge abnormal behavior, can avoid the limitations of single-feature judgment, thereby more accurately identifying abnormal behavior; further, by assigning limb weight values to different limb parts and setting weights for limb features and trajectory features respectively, key information can be highlighted according to the actual situation when judging abnormalities, effectively reducing the misjudgment rate.

[0094] In another preferred implementation manner, after determining that the behavior analysis result of the target object is abnormal behavior, the solution provided in this embodiment can also perform the following operations: determine the target abnormal type corresponding to the abnormal behavior, and one abnormal type corresponds to one warning mechanism; determine the target warning mechanism according to the target abnormal type, and start linkage alarm based on the target warning mechanism. By setting different warning mechanisms for different abnormal types in this embodiment, the alarm can be more accurately targeted at specific abnormal situations; through the pre-established warning mechanism and linkage alarm measures, intervention can be carried out at the early stage when abnormal behavior occurs, effectively preventing the expansion and deterioration of abnormal situations.

[0095] In this embodiment, the preset abnormal behavior types can include types such as armed threat, personnel conflict, malicious damage to equipment, etc., and different warning mechanisms are set for each type. For example, the warning mechanism corresponding to armed threat can be: linkage alarm and linkage emergency rescue, etc.; the warning mechanism corresponding to personnel conflict can be: notify security and turn on key monitoring videos, etc.; the warning mechanism for malicious damage to equipment can be: notify security and lock the entrance and exit, etc. The implementation manners of specific abnormal behavior types and corresponding warning mechanisms are not limited herein.

[0096] The behavior analysis method provided in this embodiment can extract the limb features of the target object in each frame of image based on the key point detection model, and can obtain detailed information such as the posture and actions of the target object. At the same time, combined with the trajectory features determined based on limb features and timestamps, it comprehensively considers the static limb state and dynamic movement trajectory of the target object, analyzes the behavior of the target object from multiple dimensions, and can accurately analyze the monitored behavior. Compared with the existing solution that relies on manual monitoring, it reduces the input of labor costs and effectively reduces the situations of false alarms and missed alarms.

[0097] Figure 3 is a schematic structural diagram of the behavior analysis device provided by an embodiment of the present application. This device is applicable to execute the behavior analysis method provided by the embodiment of the present application. As Figure 3 shown, this device may specifically include:

[0098] A video data processing module 310, configured to process the received real-time video stream data to obtain consecutive frame images;

[0099] A limb feature extraction module 320, configured to, when it is detected that the consecutive frame images contain a target object, respectively extract the limb features of the target object in each frame image based on a key point detection model;

[0100] A trajectory feature determination module 330, configured to determine the trajectory features of the target object based on the limb features in each frame image and the corresponding timestamps;

[0101] A behavior analysis module 340, configured to perform behavior analysis on the target object according to the limb features and the trajectory features.

[0102] The behavior analysis device provided by this embodiment can, by extracting the limb features of the target object in each frame image based on the key point detection model, obtain detailed information such as the posture and actions of the target object. At the same time, by combining the trajectory features determined based on the limb features and timestamps, it comprehensively considers the static limb state and dynamic movement trajectory of the target object, analyzes the behavior of the target object from multiple dimensions, and can accurately analyze the behavior of the monitored target object. Compared with the existing solution that relies on manual monitoring, it reduces the investment in labor costs and effectively reduces the situations of false alarms and missed alarms.

[0103] In one embodiment, the limb feature extraction module 320 includes a contour information acquisition unit, a key point detection unit, and a limb feature determination unit, where:

[0104] The contour information acquisition unit is configured to acquire the contour information of the target object in each frame image;

[0105] The key point detection unit is configured to respectively perform key point detection on the contour information in each frame image based on the key point detection model to obtain the key skeletal points and the corresponding coordinate information of the target object in each frame image; the key skeletal points include at least one;

[0106] The limb feature determination unit is configured to determine the limb features of the target object according to the coordinate information corresponding to each key skeletal point in each frame image.

[0107] In one embodiment, the limb feature determination unit is specifically configured to, for each frame of image, calculate the bone length information and joint angle information between adjacent bone points according to the coordinate information of the key bone points; extract action features according to the bone distance information and joint angle information respectively corresponding to the consecutive frame images; and determine the limb features of the target object according to the action features.

[0108] In one embodiment, the trajectory feature determination module 330 includes a coordinate determination unit, a timestamp association unit, and a trajectory feature determination unit, where:

[0109] The coordinate determination unit is configured to determine the center point coordinates of the target object based on the limb features in each frame of image;

[0110] The timestamp association unit is configured to associate the center point coordinates in each frame of image with the corresponding timestamp to obtain the trajectory points in the consecutive frame images;

[0111] The trajectory feature determination unit is configured to determine the trajectory features according to the trajectory points in the consecutive frame images.

[0112] In one embodiment, the trajectory feature determination unit is specifically configured to, in adjacent consecutive frame images, determine the movement displacement and movement direction of the target object according to the coordinate information of adjacent two trajectory points; determine the object speed of the target object between adjacent two trajectory points according to the movement displacement and time interval; determine the trajectory curvature according to adjacent three trajectory points; and obtain the trajectory features according to the movement displacement, the movement direction, the object speed, and the trajectory curvature.

[0113] In one embodiment, the behavior analysis module 340 includes a first numerical value determination unit, a second numerical value weighting unit, a weight acquisition unit, a numerical calculation unit, and a behavior determination unit, where:

[0114] The first numerical value determination unit is configured to determine a first similarity value corresponding to the standard limb features according to the limb weight values corresponding to different limb parts in the limb features;

[0115] The second numerical value weighting unit is configured to determine a corresponding second similarity numerical value according to the trajectory features and the standard trajectory features;

[0116] The weight acquisition unit is configured to acquire a first weight corresponding to the limb features and a second weight corresponding to the trajectory features;

[0117] The numerical calculation unit is configured to obtain the abnormal numerical value of the target object according to the first weight, the first similarity value, the second weight, and the second similarity value;

[0118] A behavior determination unit, configured to determine that the behavior analysis result of the target object is an abnormal behavior when the abnormal value exceeds a preset threshold.

[0119] In one embodiment, after determining that the behavior analysis result of the target object is the abnormal behavior, the apparatus further includes: an abnormal type determination module and a linkage alarm module, where:

[0120] The abnormal type determination module is configured to determine a target abnormal type corresponding to the abnormal behavior, and one abnormal type corresponds to one warning mechanism;

[0121] The linkage alarm module is configured to determine a target warning mechanism according to the target abnormal type, and start a linkage alarm based on the target warning mechanism.

[0122] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional module is used as an example. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described functional modules can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0123] It should be noted that the relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of the relevant data comply with the relevant laws, regulations, and standards of the relevant regions.

[0124] The embodiment of the present application further provides an electronic device, where the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the behavior analysis method according to any embodiment of the present application.

[0125] The embodiment of the present application further provides a computer-readable medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the behavior analysis method according to any embodiment of the present application when executed by a processor.

[0126] Next, refer to Figure 4 , Figure 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. It shows a schematic structural diagram of a computer system 500 of an electronic device suitable for implementing the embodiment of the present application. Figure 4The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0127] As Figure 4 shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage section 508 into the random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0128] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read from it can be installed into the storage section 508 as needed.

[0129] Specifically, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above functions defined in the system of the present application are executed.

[0130] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in this application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, and optical cable, etc., or any suitable combination of the above.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0132] The modules and / or units involved in the embodiments of the present application can be implemented in software or in hardware. The described modules and / or units can also be provided in a processor. For example, it can be described as: a processor includes a video data processing module, a limb feature extraction module, a trajectory feature determination module, and a behavior analysis module. Among them, the names of these modules do not constitute a limitation to the module itself in some cases.

[0133] As another aspect, the present application also provides a computer-readable medium. The computer-readable medium can be included in the device described in the above embodiments; it can also exist alone without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: processing the received real-time video stream data to obtain consecutive frame images; when it is detected that the consecutive frame images contain a target object, based on a key point detection model, respectively extracting the limb features of the target object in each frame image; determining the trajectory features of the target object based on the limb features in each frame image and the corresponding timestamps; and performing behavior analysis on the target object according to the limb features and the trajectory features.

[0134] According to the technical solution of this embodiment, by extracting the limb features of the target object in each frame image based on the key point detection model, information such as the posture and actions of the target object can be obtained in detail. At the same time, combined with the trajectory features determined based on the limb features and timestamps, the static limb state and dynamic movement trajectory of the target object are comprehensively considered, and the behavior of the target object is analyzed from multiple dimensions, enabling accurate analysis of the behavior of the monitored target object. Compared with the existing solution that relies on manual monitoring, it reduces the investment in labor costs and effectively reduces the situations of false alarms and missed alarms.

[0135] The above specific implementation manners do not constitute a limitation to the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present application should be included within the protection scope of the present application.

Claims

1. A behavior analysis method, characterized in that, Including: Processing the received real-time video stream data to obtain continuous frame images; When it is detected that the continuous frame images contain a target object, respectively extracting the limb features of the target object in each frame image based on a key point detection model; Determining the trajectory features of the target object based on the limb features and corresponding timestamps in each frame image; Performing behavior analysis on the target object according to the limb features and the trajectory features.

2. The behavior analysis method according to claim 1, characterized in that, The step of respectively extracting the limb features of the target object in each frame image based on the key point detection model includes: Obtaining the contour information of the target object in each frame image; Based on the key point detection model, respectively performing key point detection on the contour information in each frame image to obtain the key bone points and corresponding coordinate information of the target object in each frame image; the key bone points include at least one; Determining the limb features of the target object according to the coordinate information corresponding to each key bone point in each frame image.

3. The behavior analysis method according to claim 2, characterized in that The step of determining the limb features of the target object according to the coordinate information corresponding to each key bone point in each frame image includes: For each frame image, calculating the bone length information and joint angle information between adjacent bone points according to the coordinate information of the key bone points; Extracting action features according to the bone distance information and joint angle information corresponding to the continuous frame images; Determining the limb features of the target object according to the action features.

4. The behavior analysis method according to claim 1, characterized in that The step of determining the trajectory features of the target object based on the limb features and corresponding timestamps in each frame image includes: Determining the center point coordinates of the target object based on the limb features in each frame image; Associating the center point coordinates in each frame image with the corresponding timestamps to obtain the trajectory points in the continuous frame images; Determining the trajectory features according to the trajectory points in the continuous frame images.

5. The behavioral analysis method according to claim 4, wherein The step of determining the trajectory features according to the trajectory points in the continuous frame images includes: In adjacent continuous frame images, determining the moving displacement and motion direction of the target object according to the coordinate information of adjacent two trajectory points; Determining the object speed of the target object between adjacent two trajectory points according to the moving displacement and time interval; Determining the trajectory curvature according to adjacent three trajectory points; Obtaining the trajectory features according to the moving displacement, the motion direction, the object speed and the trajectory curvature.

6. The behavior analysis method according to claim 1, wherein The step of performing behavior analysis on the target object according to the limb features and the trajectory features includes: Determining a first similarity value corresponding to the standard limb features according to the limb weight values corresponding to different limb parts in the limb features; Determining a corresponding second similarity value according to the trajectory features and the standard trajectory features; Obtaining a first weight corresponding to the limb features and a second weight corresponding to the trajectory features; Obtaining an abnormal value of the target object according to the first weight, the first similarity value, the second weight and the second similarity value; When the abnormal value exceeds a preset threshold, determining that the behavior analysis result of the target object is that there is an abnormal behavior.

7. The behavior analysis method according to claim 6, wherein After determining that the behavior analysis result of the target object is an abnormal behavior, the method further includes: Determine the target abnormal type corresponding to the abnormal behavior, where one abnormal type corresponds to one warning mechanism; Determine the target warning mechanism according to the target abnormal type, and start the linkage alarm based on the target warning mechanism.

8. A behavior analysis device, characterized in that, It includes: A video data processing module for processing the received real-time video stream data to obtain continuous frame images; A limb feature extraction module for, when detecting that the continuous frame images contain a target object, respectively extracting the limb features of the target object in each frame image based on a key point detection model; A trajectory feature determination module for determining the trajectory features of the target object based on the limb features and the corresponding timestamps in each frame image; A behavior analysis module for performing behavior analysis on the target object according to the limb features and the trajectory features.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the behavior analysis method according to any one of claims 1-7.

10. A computer program product, including a computer program, where the computer program, when executed by a processor, implements the behavior analysis method according to any one of claims 1-7.

Images