Target marking method and device based on motion difference
Through the method based on action difference, the action deviation parameters are determined using joint angle data to mark special targets in the video, which solves the problem that special targets cannot be accurately marked in the intelligent monitoring system and improves the accuracy of marking.
Patent Information
- Application Number
- CN202111310593.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-11-05
AI Technical Summary
The existing intelligent monitoring system cannot accurately mark special targets in videos, mainly because abnormal behavior cannot be exhaustive, the probability of a single abnormal behavior is low, the sample is small, and the definition of abnormalities in different scenarios is different, resulting in the inability to accurately identify and label videos. special goals that need attention.
The target marking method based on action difference is adopted to obtain joint angle data at the target moment to determine the deviation parameters of the action change. If the maximum deviation parameter exceeds the preset threshold, the target is marked as an abnormal target to achieve the special target in the video. Mark accurately.
Improves the labeling accuracy of special targets, without pre-establishing anomaly sample library or defining positive abnormal behaviors, it can accurately identify special targets that need attention in the image.
Smart Images

Figure CN114663972B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a target marking method and device based on motion difference. Background Art
[0002] With the development of computer technology, intelligent monitoring systems based on computer vision technology, such as target detection, target tracking, and event detection, have been widely used, for example, in the fields of medical monitoring, traffic supervision, and public safety.
[0003] At present, most intelligent monitoring systems use multi-target tracking combined with abnormal behavior discrimination technology to identify special targets. Multi-target tracking is the process of finding the target of interest in the current frame in the subsequent frames of a video, focusing on all targets in the video. In actual scenarios, not only do we need to track all targets in the video, but we also need to focus on a certain target, which is a special target compared to other targets in the picture. In terms of identifying special targets, most of them use a combination of abnormal behavior recognition and abnormal behavior detection. For abnormal behavior recognition, it is necessary to establish a sample library for abnormal postures or actions, and then use human target detection, posture estimation, action recognition and other methods to identify specific behaviors, and finally determine whether they belong to the category of abnormal behavior sample library; for abnormal behavior detection, the determination of abnormal scenes in the video is mainly achieved by comparing the similarity with normal scenes. However, in actual scenarios, due to the inability to exhaustively enumerate abnormal behaviors, the low probability of occurrence of a single abnormal behavior, the small number of samples, the lack of clear definition of positive abnormal behavior, and the different definitions of abnormalities in different scenarios, it is impossible to accurately mark the special targets that need to be paid attention to in the video. Summary of the invention
[0004] In view of this, the present application provides a target marking method and device based on motion difference, the main purpose of which is to improve the technical problem in the prior art that special targets that need to be paid attention to in the video cannot be accurately marked.
[0005] According to one aspect of the present application, a target marking method based on motion difference is provided, comprising:
[0006] Obtain the joint angle data of each identified target at the target moment;
[0007] Based on the joint angle data, determining a motion deviation parameter resulting from a deviation in motion changes between the identified targets;
[0008] If the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, the identified target corresponding to the maximum motion deviation parameter is marked as an abnormal target.
[0009] Preferably, the method further comprises:
[0010] If the motion deviation parameters are all smaller than the preset motion deviation parameter thresholds, the moment deviation parameters of the deviations caused by the motion changes of each single recognition target at multiple target moments are determined based on the joint angle data, and the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters is marked as an abnormal target.
[0011] Preferably, before acquiring the joint angle data of each identified target at the target moment, the method further includes:
[0012] Acquire image information of the identified target at the target time;
[0013] Generate a joint point confidence map and a human body part affinity field of the recognition target based on a convolutional network;
[0014] According to the joint point confidence map and the human body part affinity field, the joint points of the recognition target are connected to generate original skeleton data of the recognition target.
[0015] Preferably, after generating the original skeleton data of the recognition target, the method further comprises:
[0016] Adjusting the size of each joint point in the original skeleton data to the same unit;
[0017] Screening out the joint points of the head in the original skeleton data;
[0018] Screening out the original skeleton data with missing parts, wherein the missing parts are neck joints and / or thigh joints;
[0019] The joint points in the blank part of the original skeleton data are filled to generate the skeleton data of the recognition target, and the joint points in the blank part are used to represent the joint points except the neck joint points and the thigh joint points.
[0020] Preferably, after generating the skeleton data of the recognition target, the method further comprises:
[0021] Based on the skeleton data of the recognition target, the joint angle data of the recognition target are determined.
[0022] Preferably, the method further comprises:
[0023] Based on the joint angle data of each of the identified targets at the multiple target moments, a target time matrix is generated to store the state information of each of the identified targets at the multiple target moments.
[0024] Preferably, before acquiring the image information of the identified target at the target moment, the method further includes:
[0025] Acquire the original image information at the target moment;
[0026] Based on the feature extraction network, feature extraction is performed on the original image information to obtain a feature map of a preset size to serve as image information of the identified target at the target moment.
[0027] According to another aspect of the present application, a target marking device based on motion difference is provided, comprising:
[0028] The first acquisition module is used to acquire the joint angle data of each identified target at the target moment;
[0029] A first determination module is used to determine the motion deviation parameter of the deviation caused by the motion change between the recognition targets based on the joint angle data;
[0030] The marking module is used to mark the identification target corresponding to the maximum action deviation parameter as an abnormal target if the maximum action deviation parameter among the action deviation parameters is greater than a preset action deviation parameter threshold.
[0031] Preferably, the marking module,
[0032] It is also used to determine the moment deviation parameters of the deviations caused by the movement changes of each single recognition target at multiple target moments based on the joint angle data if the movement deviation parameters are all less than the preset movement deviation parameter thresholds, and mark the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters as an abnormal target.
[0033] Preferably, before the first acquisition module, the device further includes:
[0034] A second acquisition module is used to acquire image information of the identified target at the target time;
[0035] A first generation module, used for generating a joint point confidence map and a human body part affinity field of the recognition target based on a convolutional network;
[0036] A connection module is used to connect the joint points of the recognition target according to the joint point confidence map and the human body part affinity field to generate the original skeleton data of the recognition target.
[0037] Preferably, after the connection module, the device further comprises:
[0038] An adjustment module, used for adjusting the size of each joint point in the original skeleton data to the same unit;
[0039] A screening module, used for screening out the joint points of the head in the original skeleton data;
[0040] The screening module is further used to screen out the original skeleton data with missing parts, wherein the missing parts are neck joints and / or thigh joints;
[0041] A filling module is used to fill the joint points in the blank part of the original skeleton data to generate the skeleton data of the recognition target, and the joint points in the blank part are used to represent the joint points except the neck joint points and the thigh joint points.
[0042] Preferably, after the filling module, the device further comprises:
[0043] The second determination module is used to determine the joint angle data of the identified target based on the skeleton data of the identified target.
[0044] Preferably, the device further comprises:
[0045] The second generating module is used to generate a target time matrix based on the joint angle data of each identified target at the multiple target moments, so as to store the state information of each identified target at the multiple target moments.
[0046] Preferably, before the second acquisition module, the device further includes:
[0047] A third acquisition module is used to acquire the original image information at the target moment;
[0048] The extraction module is used to extract features from the original image information based on a feature extraction network to obtain a feature map of a preset size as the image information of the identified target at the target moment.
[0049] According to another aspect of the present application, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned target marking method based on motion differentiation.
[0050] According to another aspect of the present application, a terminal is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;
[0051] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned target marking method based on motion difference.
[0052] By means of the above technical solution, the technical solution provided by the embodiment of the present application has at least the following advantages:
[0053] The present application provides a target marking method and device based on motion difference, which first obtains the joint angle data of each identified target at the target moment; secondly, based on the joint angle data, determines the motion deviation parameters of the deviations caused by the motion changes between the identified targets; finally, if the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, marks the identified target corresponding to the maximum motion deviation parameter as an abnormal target. Compared with the prior art, the embodiment of the present application determines the motion deviation parameters of the identified targets in the image, marks the identified targets with the maximum motion deviation parameters and exceeding the threshold as abnormal targets, thereby achieving the purpose of accurately marking special targets that need to be paid attention to in the image and improving the accuracy of special target marking.
[0054] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0056] Figure 1 A flow chart of a target marking method based on action difference provided in an embodiment of the present application is shown;
[0057] Figure 2 A flow chart of another target marking method based on action difference provided in an embodiment of the present application is shown;
[0058] Figure 3 A schematic diagram of original skeleton data provided by an embodiment of the present application is shown;
[0059] Figure 4 A schematic diagram of skeleton data for removing head joint points provided by an embodiment of the present application is shown;
[0060] Figure 5 A schematic diagram of joint angles provided by an embodiment of the present application is shown;
[0061] Figure 6 A block diagram of a target marking device based on motion difference provided in an embodiment of the present application is shown;
[0062] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0063] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0064] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0065] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.
[0066] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0067] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0068] The embodiments of the present application can be applied to computer systems / servers, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0069] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.
[0070] At present, the identification of abnormal behaviors of targets in videos has been widely used in society and production. For example, in the field of medical monitoring, through the abnormal behavior recognition and detection technology, real-time monitoring of unattended patients or elderly people can be achieved, judging whether the target has fallen or other accidents, and calling the police for help in time to ensure that they receive treatment and help as soon as possible; in the field of traffic supervision, relevant departments can use the human abnormal behavior recognition and detection technology to monitor abnormal situations inside and outside the car, including driver behavior, to reduce the risk of accidents; in the field of public safety, this technology can be used to detect abnormal situations of people in public places and judge violations of social security.
[0071] Existing technologies mainly focus on abnormal behavior recognition and abnormal behavior detection. For abnormal behavior recognition, firstly, abnormal behaviors and postures are divided, and an abnormal behavior sample library is established. The specific behavior is identified using action recognition methods (human target detection, posture estimation, action recognition, etc.), that is, the event or behavior of interest is defined in advance, and the network is trained with labels. By identifying the specific actions and postures of the target, it is determined whether its behavior is abnormal, and finally whether it belongs to the category of abnormal behavior sample library. For anomaly detection, specific abnormal actions are less considered, and experience is often learned from a large amount of video data, and behaviors are divided into two categories: normal and abnormal. The abnormal scene in the video is determined by comparing the similarity with the normal scene.
[0072] However, in actual applications, there are a few examples of abnormal behavior for a single target in abnormal behavior identification; in crowded scenes, it is costly to label each behavior, and there is no guarantee that the labeling can cover all abnormal behaviors; and there are many types of abnormal behaviors, which cannot be exhaustively listed. In actual applications, there are problems such as unclear definitions of positive and abnormal behaviors; different definitions of abnormalities in different scenarios, making it difficult for abnormal behavior detection systems to generalize; and the probability of abnormal behavior is much lower than that of normal behavior, making it difficult to learn enough abnormal features.
[0073] In view of the above problems, the present application provides a target marking method based on action difference, such as Figure 1 As shown, the method includes:
[0074] 101. Obtain joint angle data of each identified target at the target moment.
[0075] In an embodiment of the present application, the joint angle data of each recognition target in the target moment image is first obtained. Wherein, the target moment is the selected moment for recognition, which can be a random moment in the real-time monitoring video, or a pre-set fixed moment (for example, setting the time for physical exercises between classes, etc.), which is not specifically limited in the embodiment of the present application. The recognition target is a person or object in the image, for example, a patient being monitored in the field of medical monitoring, or a vehicle or driver in the field of traffic supervision, etc., which is not specifically limited in the embodiment of the present application. The joint angle data is used to characterize the motion parameters of the recognition target at the target moment, and may include the recognition target joint angle, velocity, acceleration, etc.
[0076] It should be noted that the motion parameters of the current recognition target can be determined through the joint angle data, and the abnormal target whose motion is most different from other recognition targets can be marked by comparing the motion parameters of each recognition target. Compared with the existing technology, aszx does not need to establish an abnormal behavior sample library in advance, nor does it need to define positive abnormal behaviors in advance. It can accurately identify special targets that need attention in video images, thus overcoming the defects in the existing technology.
[0077] 102. Based on the joint angle data, determine the motion deviation parameters of the deviations caused by the motion changes between the recognition targets.
[0078] In the embodiment of the present application, the motion deviation parameters of the deviations caused by the motion changes between the various recognition targets are determined based on the joint angle data of the recognition target obtained in step 101. The motion deviation parameters are used to characterize the motion deviation between the current recognition target and other recognition targets in the image. Since the motion deviation is difficult to define and quantify, the motion of the recognition target is quantified based on the data such as the joint angle, velocity, acceleration, etc. of the recognition target to form motion deviation parameters that can be numerically compared, so as to determine the abnormal target with the largest motion deviation from other recognition targets and exceeding the preset motion deviation parameter threshold.
[0079] 103. If the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, the identified target corresponding to the maximum motion deviation parameter is marked as an abnormal target.
[0080] In an embodiment of the present application, when the maximum motion deviation parameter among the motion deviation parameters between each identified target and other identified targets determined by step 102 is greater than the preset motion deviation parameter threshold, it means that it is a special target different from other identified targets. At this time, it can be identified as an abnormal target and added to the tracking target set for target tracking.
[0081] The present application embodiment provides another target marking method based on action difference, such as Figure 2 As shown, the method includes:
[0082] 201. Obtain original image information at a target time.
[0083] In an embodiment of the present application, a video frame image at a target moment is first selected as original image information based on a surveillance video. The target moment may be a random moment in a real-time surveillance video or a pre-set fixed moment, which is not specifically limited in the embodiment of the present application.
[0084] It is understandable that the embodiment of the present application can decompose a video into individual images in units of video frames, and by selecting a target moment, the original image information of that moment can be obtained.
[0085] 202. Based on the feature extraction network, extract features from the original image information to obtain a feature map of a preset size as image information of the identified target at the target moment.
[0086] In the embodiment of the present application, in order to avoid background interference with the detection results, feature extraction can be performed on the original image information based on the feature extraction network to obtain a feature map, which only contains all the recognition targets of interest as the input image for marking abnormal targets, and further determines the motion deviation parameters of each recognition target. Specifically, the candidate box can be selected first and then deep learning can be used for image feature extraction, such as the R-CNN algorithm, or the target detection framework of deep learning regression can be directly used for image feature extraction, such as the SSD, YOLO algorithm, etc., which is not specifically limited in the embodiment of the present application.
[0087] Exemplarily, taking the YOLO algorithm as an example, the feature extraction network is first used to extract features from the original image information to obtain a feature map output of a specific size. First, the original image information is divided into grid units, for example, 13×13 grid units, or it can be divided more finely or slightly roughly according to the specific situation of the image. If the center coordinates of an object in the real frame fall within a grid unit, then the grid unit is used to predict the object. Each target has a fixed number of bounding boxes. For example, there are three bounding boxes in YOLO v3. Logistic regression is used to determine the regression box used for prediction. The detection of the target uses a multi-scale prediction method to improve the detection accuracy of small targets. The classifier can be changed from softmax to logistic to improve the target overlap. Since YOLOv3 is an end-to-end target detection method, it runs very fast. Input the original image information sequence F = {f t ,f t+1 ,...,f t+m}, output recognition target image set I = {I t ,I t+1 ,...,I t+m} and the identified target location set Obj = {Obj t ,Obj t+1 ,...,Obj t+m}. The set of all n recognition target images at time t is The location set is in, Represents the coordinates of the geometric center point of the rectangular bounding box of target i at time t.
[0088] 203. Obtain image information of the identified target at a target time.
[0089] In the embodiment of the present application, a feature map obtained by feature extraction in step 202 is obtained as image information of the target to be identified at the target time, that is, an input image to be marked as an abnormal target.
[0090] 204、Generate joint point confidence map and human body part affinity field of recognition target based on convolutional network.
[0091] In the embodiment of the present application, the joint point confidence map and the human body part affinity field of each recognition target in the image information are first generated through a convolutional network. Among them, the joint point confidence map and the human body part affinity field are used to characterize the important parameters for generating the original skeleton data of the recognition target.
[0092] 205. According to the joint point confidence map and the human body part affinity field, the joint points of the recognition target are connected to generate the original skeleton data of the recognition target.
[0093] In the embodiment of the present application, for example, the joints of a single recognition object can be connected using the pair matching in graph theory. Input the recognition object image set I obtained by feature extraction = {I t ,I t+1 ,...,I t+m}, output the joint point set P of the recognition target = {P t ,P t+1 ,...,P t+m}, where the set of joint points of all n recognized targets at time t is in, represents the set of joint points for identifying target i at time t. Represents the coordinates of joint point j of the recognition target i.
[0094] Furthermore, the original skeleton data of the recognition target is generated. In a specific application scenario, as shown in Figure 3, the corresponding positions of each joint are as follows:
[0095] / / {0,"Nose"},
[0096] / / {1,"Neck"},
[0097] / / {2,"RShoulder"},
[0098] / / {3,"RElbow"},
[0099] / / {4,"RWrist"},
[0100] / / {5,"LShoulder"},
[0101] / / {6,"LElbow"},
[0102] / / {7,"LWrist"},
[0103] / / {8,"RHip"},
[0104] / / {9,"RKnee"},
[0105] / / {10,"RAnkle"},
[0106] / / {11,"LHip"},
[0107] / / {12,"LKnee"},
[0108] / / {13,"LAnkle"},
[0109] / / {14,"REye"},
[0110] / / {15,"LEye"},
[0111] / / {16,"REar"},
[0112] / / {17,"LEar"}
[0113] 206. Adjust the size of each joint point in the original skeleton data to the same unit.
[0114] In the embodiment of the present application, in order to optimize the data processing process, the original skeleton data of the recognition target is preprocessed before determining the joint angle data. First, each joint position is scaled to the same unit to process images with different height / width ratios.
[0115] 207. Filter out the joint points of the head in the original skeleton data.
[0116] In the embodiment of the present application, in the original skeleton data, the head has 5 joints, including 1 head, 2 eyes and 2 ears. However, for action recognition based on joint angle data, these five joints play a very small role. What is important is the configuration of the body and limbs. Therefore, the five joints of the head can be screened out to make the function more meaningful. In actual application scenarios, the skeleton data with the head joints screened out is as follows: Figure 4 Shown
[0117] 208. Filter out the original skeleton data with missing parts.
[0118] The missing parts are the neck joints and / or thigh joints;
[0119] In the embodiment of the present application, if the skeleton of the recognition target is not detected, or the detected skeleton has no neck or thighs, the frame is considered invalid and is screened out. In addition, it should be reinitialized in the next frame.
[0120] 209. Fill in the blank joint points in the original skeleton data to generate skeleton data for the recognition target.
[0121] Among them, the joint points in the blank part are used to represent the joint points except the neck joint points and the thigh joint points.
[0122] In the embodiment of the present application, there may be a situation where joint points are missing in actual application scenarios, and the complete skeleton data of the recognition target may not be detected from the image, resulting in some gaps in the joint positions. These joints must be filled in order to maintain a fixed-size feature vector for the feature classification process.
[0123] For example, fill in the joint positions based on the relative position of the current recognition target with respect to the neck in the previous frame. Suppose in the last frame, the hand was 10 pixels to the right of the neck. Then in this frame, if the hand is missing, set it to 10 pixels to the right of the neck in the current frame. Use (x lost ,y lost ) represents the position of a missing joint in the current frame, (x curr ,y curr ) represents the filling value used to fill the missing joint position, (x prev ,y prev ) represents the position of the joint in the previous frame, see the formula:
[0124] x curr =xNeck curr +(x prev -xNeck prev )
[0125] y curr =yNeck curr+(y prev -yNeck prev )
[0126] 210. Based on the skeleton data of the identified target, determine the angle data of each joint of the identified target.
[0127] In the embodiment of the present application, based on the skeleton data obtained in the above steps, the angle data of each joint of the recognition target is determined.
[0128] For example, the angle of the joint point is Figure 5 As shown, the vector formed by point 0 and point 1 can be expressed as v1 = (x1-x0, y1-y0), and the vector formed by point 0 and point 2 can be expressed as v2 = (x2-x0, y2-y0). Calculate the angle of the joint point:
[0129]
[0130] Based on the skeleton data obtained in the above steps, a total of 8 required joint angles are defined as follows:
[0131] θ (2,3,4) ,θ (5,6,7) ,θ (1,2,3) ,θ (1,5,6) ,θ (8,9,10) ,θ (11,12,13) ,θ (1,8,9) ,θ (1,11,12)
[0132] 211. Based on the joint angle data, determine the motion deviation parameters of the deviations caused by the motion changes between the recognition targets.
[0133] In the embodiment of the present application, when the number of recognition targets n>1, postureBias is used i t(i∈n) represents the posture deviation of the identified target i at time t from other identified targets, which is calculated by weighted summation of the angle deviations of each joint point of the identified target i, as shown in the formula:
[0134]
[0135] where w j is the weight value of joint point j, obtained by principal component analysis. Specifically, assuming that the coefficient values of the i-th sample are denoted as [e j1 ,e j2 ,...,e jp ], from which we can construct the matrix A=(e ji ) p×p , where p represents the number of influencing factors. [c1,c2,...,c p] is the influence weight coefficient of the total value of the first p influencing factors of i training samples obtained by principal component analysis, which can be used as the weight parameter of other subsequent identification samples. The other identification samples are obtained by formula w j =c1·e j1 +c2·e j2 +...+c p ·e jp , and get the final fusion score result w j .
[0136] postureBias ij It represents the angle deviation of joint point j of target i, and the calculation method is shown in the formula:
[0137]
[0138] in Represents the set of angles of n identified target joint points j.
[0139] Calculate the rate deviation between the identified target i and other identified targets at time t:
[0140]
[0141] in Represents the rate set of n identified targets at time t.
[0142] Calculate the acceleration deviation between the identified target i and other identified targets at time t:
[0143]
[0144] in Represents the acceleration set of n identified targets at time t.
[0145] The motion deviation parameter of the identified target i at time t and other identified targets is obtained by weighted summation of posture deviation, velocity deviation and acceleration deviation: Get the action deviation parameter set at time t Among them, the influence weights of the three deviations are obtained by principal component analysis.
[0146] It should be noted that when the number of identified targets n = 1, i (i∈n) is directly added to the tracking object set T t .
[0147] 212a. If the maximum motion deviation parameter among the motion deviation parameters is greater than the preset motion deviation parameter threshold, the identified target corresponding to the maximum motion deviation parameter is marked as an abnormal target.
[0148] In the embodiment of the present application, the B obtained from step 211 tTake the maximum deviation b i t If b i t >Preset action deviation parameter threshold, it means that the identified target corresponding to the maximum value is a special target in the image, and it is marked as an abnormal target to be added to the tracking object set T t If T t If it is empty, it means that all elements in the motion deviation parameter set are less than the preset motion deviation parameter threshold. Furthermore, the motion deviation parameter threshold can be pre-set based on business needs to measure whether the motion deviation parameters of multiple identified targets at the same time are within an acceptable range in the current scenario, that is, the range of abnormal motion amplitudes allowed.
[0149] 212b. If the motion deviation parameters are all less than the preset motion deviation parameter threshold, then based on the joint angle data, determine the moment deviation parameters of the deviations caused by the motion changes of each single recognition target at multiple target moments, and mark the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters as an abnormal target.
[0150] In the embodiment of the present application, if the action deviation parameters are all less than the preset action deviation parameter threshold, it means that at the current target moment, the actions of the various recognition targets in the image are not very different, and it is difficult to mark abnormal targets. At this time, the moment deviation parameters of each single recognition target at multiple target moments can be determined based on the joint angle data, and the recognition target corresponding to the maximum value of the moment deviation parameter is marked as an abnormal target, that is, the actions of the same recognition target at different moments are compared vertically. Among them, the moment deviation parameter is used to characterize the posture deviation of the same recognition target at different moments.
[0151] Specifically, use It represents the posture deviation between the identified target i at time t and the identified target at time t-1, which is obtained by weighted calculation of the angle deviation of each joint point of the identified target i, as shown in the formula:
[0152]
[0153] where w j is the weight value of joint point j, postureTimeBias ij It represents the angle deviation of joint point j of target i, and the calculation method is shown in the formula:
[0154]
[0155] in represents the angle of joint point j of the recognition target i at time t, Represents the angle of joint point j of the identified target i at time t-1.
[0156] Calculate the rate deviation of the identified target i at time t and at time t-1:
[0157]
[0158] in represents the rate of identifying target i at time t, Represents the rate of identifying target i at time t-1.
[0159] Calculate the acceleration deviation between the identified target i and other identified targets at time t:
[0160]
[0161] in represents the acceleration of the identified target i at time t, Represents the acceleration of the identified target i at time t-1.
[0162] The moment deviation parameters of object i at time t and time t-1 are obtained by weighted calculation of attitude deviation, velocity deviation and acceleration deviation. Get the time deviation parameter set at time t
[0163] From the set C t The maximum deviation is Add the identified target i to the tracking object set T t .
[0164] In order to more conveniently and formally illustrate the motion deviation parameters between each recognition target and the moment deviation parameters of a single recognition target at multiple target moments, the method of the embodiment of the present application also includes: generating a target time matrix based on the joint angle data of each recognition target at multiple target moments to store the state information of each recognition target at multiple target moments.
[0165] For example, suppose the target detection module obtains n identified targets {1, 2, ..., n} on the video frame image at time t, then the state set of all n identified targets at time t is Furthermore, the state set of all n recognition targets from the current time t to the end time t+m can be expressed as in, Represents the state sequence of the recognition target i from t to t+m. Among them, The storage information is shown in the formula:
[0166]
[0167] in,
[0168] in, To identify the angle of the jth joint point of target i, To identify the velocity of target i at time t, To identify the acceleration of target i at time t. and Calculated in the target detection process, see formula:
[0169]
[0170]
[0171] in To identify the geometric center coordinates of the rectangular bounding box of target i at time t, is the geometric center coordinate of the rectangular bounding box of the identified target i at time t-1.
[0172] The target time matrix is established based on the state set as follows:
[0173]
[0174] The present application provides a target marking method based on motion difference, which first obtains the joint angle data of each identified target at the target moment; secondly, based on the joint angle data, determines the motion deviation parameters of the deviations caused by the motion changes between the identified targets; finally, if the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, marks the identified target corresponding to the maximum motion deviation parameter as an abnormal target. Compared with the prior art, the embodiment of the present application determines the motion deviation parameters of the identified targets in the image, marks the identified targets with the maximum motion deviation parameters and exceeding the threshold as abnormal targets, thereby achieving the purpose of accurately marking special targets that need attention in the image and improving the accuracy of special target marking.
[0175] Furthermore, as a response to the above Figure 1 The implementation of the method shown in the embodiment of the present application provides a target marking device based on action difference, such as Figure 6 As shown, the device comprises:
[0176] A first acquisition module 31 , a first determination module 32 , and a marking module 33 .
[0177] The first acquisition module 31 is used to acquire the joint angle data of each identified target at the target moment;
[0178] A first determination module 32, for determining, based on the joint angle data, a motion deviation parameter resulting from a deviation in motion changes between the recognition targets;
[0179] The marking module 33 is used to mark the identification target corresponding to the maximum action deviation parameter as an abnormal target if the maximum action deviation parameter among the action deviation parameters is greater than a preset action deviation parameter threshold.
[0180] In a specific application scenario, the marking module,
[0181] It is also used to determine the moment deviation parameters of the deviations caused by the movement changes of each single recognition target at multiple target moments based on the joint angle data if the movement deviation parameters are all less than the preset movement deviation parameter thresholds, and mark the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters as an abnormal target.
[0182] In a specific application scenario, before the first acquisition module, the device further includes:
[0183] A second acquisition module is used to acquire image information of the identified target at the target time;
[0184] A first generation module, used for generating a joint point confidence map and a human body part affinity field of the recognition target based on a convolutional network;
[0185] A connection module is used to connect the joint points of the recognition target according to the joint point confidence map and the human body part affinity field to generate the original skeleton data of the recognition target.
[0186] In a specific application scenario, after the connection module, the device further includes:
[0187] An adjustment module, used for adjusting the size of each joint point in the original skeleton data to the same unit;
[0188] A screening module, used for screening out the joint points of the head in the original skeleton data;
[0189] The screening module is further used to screen out the original skeleton data with missing parts, wherein the missing parts are neck joints and / or thigh joints;
[0190] A filling module is used to fill the joint points in the blank part of the original skeleton data to generate the skeleton data of the recognition target, and the joint points in the blank part are used to represent the joint points except the neck joint points and the thigh joint points.
[0191] In a specific application scenario, after the filling module, the device further includes:
[0192] The second determination module is used to determine the joint angle data of the identified target based on the skeleton data of the identified target.
[0193] In a specific application scenario, the device further includes:
[0194] The second generating module is used to generate a target time matrix based on the joint angle data of each identified target at the multiple target moments, so as to store the state information of each identified target at the multiple target moments.
[0195] In a specific application scenario, before the second acquisition module, the device further includes:
[0196] A third acquisition module is used to acquire the original image information at the target moment;
[0197] The extraction module is used to extract features from the original image information based on a feature extraction network to obtain a feature map of a preset size as the image information of the identified target at the target moment.
[0198] The present application provides a target marking device based on motion difference, which first obtains the joint angle data of each identified target at the target moment; secondly, based on the joint angle data, determines the motion deviation parameters of the deviations caused by the motion changes between the identified targets; finally, if the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, marks the identified target corresponding to the maximum motion deviation parameter as an abnormal target. Compared with the prior art, the embodiment of the present application determines the motion deviation parameters of the identified targets in the image, marks the identified targets with the maximum motion deviation parameters and exceeding the threshold as abnormal targets, thereby achieving the purpose of accurately marking special targets that need attention in the image and improving the accuracy of special target marking.
[0199] According to an embodiment of the present application, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the target marking method based on motion difference in any of the above method embodiments.
[0200] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.
[0201] Figure 7 A schematic diagram of the structure of a terminal provided according to an embodiment of the present application is shown, and the specific embodiment of the present application does not limit the specific implementation of the terminal.
[0202] like Figure 7As shown, the terminal may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .
[0203] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .
[0204] The communication interface 404 is used to communicate with other devices such as clients or other servers.
[0205] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the embodiment of the target marking method based on action differentiation of the above interface.
[0206] Specifically, the program 410 may include program codes, which include computer operation instructions.
[0207] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the terminal may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0208] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0209] The program 410 may be specifically configured to enable the processor 402 to perform the following operations:
[0210] Obtain the joint angle data of each identified target at the target moment;
[0211] Based on the joint angle data, determining a motion deviation parameter resulting from a deviation in motion changes between the identified targets;
[0212] If the maximum motion deviation parameter among the motion deviation parameters is greater than a preset motion deviation parameter threshold, the identified target corresponding to the maximum motion deviation parameter is marked as an abnormal target.
[0213] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device based on the action differential target tag, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
[0214] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0215] The method and apparatus of the present application may be implemented in many ways. For example, the method and apparatus of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present application may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.
[0216] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.
[0217] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A target marking method based on motion difference, characterized in that: include: Obtain the joint angle data of each identified target at the target moment; Based on the joint angle data, determining a motion deviation parameter resulting from a deviation in motion changes between the identified targets; If the maximum action deviation parameter among the action deviation parameters is greater than a preset action deviation parameter threshold, marking the identified target corresponding to the maximum action deviation parameter as an abnormal target; If the motion deviation parameters are all smaller than the preset motion deviation parameter thresholds, the moment deviation parameters of the deviations caused by the motion changes of each single recognition target at multiple target moments are determined based on the joint angle data, and the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters is marked as an abnormal target.
2. The method according to claim 1, characterized in that Before obtaining the joint angle data of each identified target at the target moment, the method further includes: Acquire image information of the identified target at the target time; Generate a joint point confidence map and a human body part affinity field of the recognition target based on a convolutional network; According to the joint point confidence map and the human body part affinity field, the joint points of the recognition target are connected to generate original skeleton data of the recognition target.
3. The method according to claim 2, characterized in that After generating the original skeleton data of the recognition target, the method further includes: Adjusting the size of each joint point in the original skeleton data to the same unit; Screening out the joint points of the head in the original skeleton data; Screening out the original skeleton data with missing parts, wherein the missing parts are neck joints and / or thigh joints; The joint points in the blank part of the original skeleton data are filled to generate the skeleton data of the recognition target, and the joint points in the blank part are used to represent the joint points except the neck joint points and the thigh joint points.
4. The method according to claim 3, characterized in that After generating the skeleton data of the recognition target, the method further includes: Based on the skeleton data of the recognition target, the joint angle data of the recognition target are determined.
5. The method according to claim 1, characterized in that The method further comprises: Based on the joint angle data of each of the identified targets at the multiple target moments, a target time matrix is generated to store the state information of each of the identified targets at the multiple target moments.
6. The method according to claim 2, characterized in that Before acquiring the image information of the identified target at the target time, the method further includes: Acquire the original image information at the target moment; Based on the feature extraction network, feature extraction is performed on the original image information to obtain a feature map of a preset size to serve as image information of the identified target at the target moment.
7. A target marking device based on motion difference, characterized in that: include: The first acquisition module is used to acquire the joint angle data of each identified target at the target moment; A first determination module is used to determine the motion deviation parameter of the deviation caused by the motion change between the recognition targets based on the joint angle data; a marking module, configured to mark the identified target corresponding to the maximum action deviation parameter as an abnormal target if the maximum action deviation parameter among the action deviation parameters is greater than a preset action deviation parameter threshold; The marking module is also used to determine the moment deviation parameters of the deviations caused by the movement changes of each single recognition target at multiple target moments based on the joint angle data if the movement deviation parameters are all less than the preset movement deviation parameter thresholds, and mark the recognition target corresponding to the maximum moment deviation parameter among the moment deviation parameters as an abnormal target.
8. A storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the target marking method based on motion difference as described in any one of claims 1 to 6.
9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the target marking method based on motion difference as described in any one of claims 1-6.
