A human and object based key point spatio-temporal matching degree based behavior recognition method

Abstract

The application provides a human and object key point space-time matching degree-based behavior recognition method, including the following steps: S1, pre-processing an input video; S2, converting space-time information of human body actions into human body key point and time sequence information of mutual objects; S3, constructing a space-time matrix from time sequence information of a fixed time window; S4, calculating similarity between the space-time matrix and a time sequence template; S5, selecting a time sequence template with the highest similarity, and the action corresponding to the time sequence template is the behavior action of the space-time matrix. The application can fully combine the interaction relationship between humans and objects in different scenes, convert the human-object interaction relationship into features required by a model, and effectively improve the behavior recognition effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a behavior recognition method based on the spatiotemporal matching degree of key points of people and objects. Background Technology

[0002] Existing behavior recognition technologies primarily rely on video analytics to identify behaviors by extracting human actions or posture features. Commonly used methods include convolutional neural networks (CNNs) for spatial feature extraction, recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) for processing temporal information, and pose estimation methods based on human keypoint detection. These methods are widely used in surveillance, smart homes, sports analytics, and other fields, and can accurately identify basic behavioral patterns. However, most existing methods focus on analyzing human posture or actions while neglecting the relative relationships between the human body and surrounding objects. They lack effective spatiotemporal information modeling and cannot fully capture the impact of object-human interactions on behavior.

[0003] While existing technologies have achieved some success in certain scenarios, they still have certain shortcomings. First, existing methods are easily affected by interference in high-noise environments, leading to a decrease in the accuracy of behavior recognition. Second, the lack of sufficient modeling of the relative position and motion of objects and the human body results in poor performance in recognizing complex interactive behaviors. Furthermore, existing technologies generally lack standardized action library support, making it difficult to achieve high-precision action classification in certain specific application scenarios. Therefore, a new technical solution is urgently needed to effectively address these issues and improve behavior recognition capabilities in complex scenarios. Summary of the Invention

[0004] This invention aims to at least solve the technical problem of low behavior recognition accuracy caused by ignoring the correlation between interactive objects and human behavior in the prior art. In particular, it innovatively proposes a behavior recognition method based on the spatiotemporal matching degree of key points between people and objects.

[0005] To achieve the above-mentioned objectives of the present invention, the present invention provides a behavior recognition method based on the spatiotemporal matching degree of key points of people and objects, comprising the following steps:

[0006] S1, Preprocess the input video;

[0007] S2 converts the spatiotemporal information of human body movements into the temporal information of human body key points and interactive objects;

[0008] S3, constructs a spatiotemporal matrix from the temporal information of a fixed time window;

[0009] S4, calculate the similarity between the spatiotemporal matrix and the time series template;

[0010] S5. Select the time sequence template with the highest similarity. The action corresponding to the time sequence template is the behavior action of the spatiotemporal matrix.

[0011] Preferably, the specific steps of step S1 are as follows:

[0012] S11 processes the input video into a single-frame image using OpenCV.

[0013] S12 performs noise reduction and filtering on the image processed by S11;

[0014] S13: The preprocessed image is center-cropped and resized to the required dimensions to standardize the image size.

[0015] Preferably, the specific steps of step S2 are as follows:

[0016] S21, obtain the coordinates of various key points of the human body and the coordinates of interactive objects from the preprocessed video; the coordinates of the human body key points are obtained through the YOLO-Pose model, and the coordinates of the interactive objects are obtained through the YOLO object detection model.

[0017] S22, calculate the relative distance between each key point and the interactive object. The relative distance is calculated using Euclidean distance. Store the calculated distance for later use.

[0018] Preferably, step S2 further includes:

[0019] If multiple human bodies are detected, the person with the highest confidence level is selected as the target.

[0020] If multiple interactive objects are detected for the target, the interactive object with the highest confidence level is selected.

[0021] Preferably, the maximum confidence level includes:

[0022] For the confidence level of a person, the confidence level is first calculated using formula (1). If the confidence level exceeds the set threshold, the person is identified as a person. Then, the confidence level of the key point is calculated using formula (2), and the maximum confidence level is selected from the results.

[0023] For interactive objects, the confidence level is calculated directly using formula (1), and the highest confidence level is selected from the results.

[0024] Formula (1) is as follows:

[0025] Conf = P(object) * IoU(pred,gt)

[0026] Where Conf represents the confidence level of each target box;

[0027] P(object) is the probability that an object exists within the frame;

[0028] IoU(pred,gt) is the maximum IoU value between the predicted box and all ground truth boxes;

[0029] Formula (2) is as follows:

[0030]

[0031] Where σ is the Sigmoid function, used to normalize the probability of the existence of keypoints;

[0032] This represents the predicted value of the key points output by the network, which can be obtained as a value between 0 and 1 using the Sigmoid function;

[0033] Preferably, the specific steps of step S3 are as follows:

[0034] S31, the relative distances between different key points and interactive objects are different sequences, which are used as columns of the matrix;

[0035] S32, for information with a sequence length greater than the fixed time window, equal-interval sampling is used, and for information with a sequence length less than the fixed time window, linear interpolation is used to meet the set time window length requirement;

[0036] S33. Combine the data processed in step S32 to obtain a spatiotemporal matrix, where the rows of the spatiotemporal matrix represent time frames and the columns of the spatiotemporal matrix represent the distances of different key points to the interactive object in different time frames.

[0037] Preferably, the specific steps of step S4 are as follows:

[0038] S41, Based on standard actions, generate a timing template corresponding to each action. The timing template is in matrix form, where rows represent time frames and columns represent the distances of different key points to the interactive object in different time frames. These timing templates contain spatiotemporal trajectory information of human key points and interactive objects.

[0039] S42, the temporal template and the spatiotemporal matrix are convolved through the same filter to obtain the feature signals of the temporal template and the spatiotemporal matrix;

[0040] S43, Calculate the norm of two characteristic signals;

[0041] S44, normalize the two characteristic signals after norm calculation, and then calculate the cross-correlation between the two signals; preferably, the formula for calculating the cross-correlation is:

[0042]

[0043] Where r(i) represents the cross-correlation of the signals;

[0044] n is the signal length;

[0045] x(i) represents the value of the normalized timing template signal when the signal position is i;

[0046] y(t+i) represents the value of the normalized spatiotemporal matrix signal at signal position t+i;

[0047] t represents the time delay between signals x(i) and y(i);

[0048] i is the index. It iterates through the samples of x(i) and y(i). For each t value, i starts from the starting index of the signal and continues until the ending position of the signal. The maximum value of t cannot exceed n-1. For actions with long durations, the t value should be larger, and vice versa.

[0049] Preferably, the key points of the human body are human joints. Key points of the human body generally refer to joints, but may also be other types of points defined by the user.

[0050] In summary, by employing the aforementioned technical solutions, this invention utilizes a purely mathematical calculation method for human behavior recognition. Compared to using neural network models, this method offers advantages such as optimized computational processes, reduced processing latency, increased computational speed, and reduced computational resources. Furthermore, human body key points are not limited to specific nodes but encompass a wider range of features. Moreover, this invention fully leverages the correlation between interactive objects and actions, converting the human-object interaction into features required by the model, thereby significantly improving the accuracy of action recognition. Specifically, by comparing with standard action data, this invention can accurately classify actions based on the degree of matching between the current action and the standard action. This approach not only enhances the stability of action recognition but also effectively avoids misjudgments and fuzzy recognition. Compared to traditional methods, this invention no longer relies solely on model training data for judgment but introduces a standard action library, making the recognition of unknown actions more reliable. This method can deeply analyze the relative motion of the human body and objects, providing a more accurate basis for action recognition. In complex scenarios, this invention can accurately capture subtle motion differences, further improving the accuracy of action recognition. Furthermore, by employing a spatiotemporal matrix construction method, the model can flexibly adapt to different input data and scene changes, effectively recognizing actions even under complex backgrounds, varying lighting conditions, and interference from dynamic objects. This adaptability and robustness compensate for the shortcomings of existing models in diverse scenarios, giving this invention a significant advantage in the field of action recognition.

[0051] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0053] Figure 1 This is a flowchart of the present invention.

[0054] Figure 2 This is the result of data processing in this case. Detailed Implementation

[0055] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0056] This invention first preprocesses the input video to improve the accuracy of subsequent model recognition. Then, it extracts the corresponding coordinates of the key points identified in each frame, calculates and records the coordinates of the key points, records the coordinates of the objects, calculates and counts the distance between the interactive objects and the center point of the human body, uses the counted distance as a spatiotemporal matrix, and calculates the matching degree between the current action and the standard action by comparing it with the obtained standard action data, and then classifies the action to obtain the result of the current behavior action recognition.

[0057] This invention proposes a behavior recognition method based on the spatiotemporal matching degree of key points between people and objects, such as... Figure 1 As shown, it includes the following steps:

[0058] S1, Preprocess the input video;

[0059] S11 processes the input video into a single-frame image using OpenCV.

[0060] S12 performs noise reduction and filtering on the image processed in S11.

[0061] S13. Crop the preprocessed image from the center and resize it to the required size to standardize the image dimensions.

[0062] S2 converts the spatiotemporal information of human body movements into temporal information of key points on the human body and interacting objects.

[0063] S21, obtain the coordinates of various key points of the human body and the coordinates of interactive objects from the preprocessed video; the coordinates of the human body key points are obtained through the YOLO-Pose model, and the coordinates of the interactive objects are obtained through the YOLO object detection model.

[0064] Specifically, the keypoint detection model outputs the sequence of skeleton keypoints of the detected person in the video in a pre-defined order, sorted by N, T, V, and C. It also outputs the recognition results of human behavior. N represents the maximum number of objects detected in the video, T represents the video frame length, V represents the number of keypoints, and C represents the number of channels (x-axis, y-axis, and confidence score). In this invention, considering that people interact not only with interactive objects in different scenes but also with other people, N is set to 2. If there is only one person in the video or only one person can be detected, and the other object cannot be detected, then 0 is output at the corresponding position of the keypoint.

[0065] This involves the issue of primary and secondary targets. When two or more people are detected simultaneously, how should the selection be made? The solution adopted in this invention is to sort them according to their confidence level. The person with the highest confidence level is designated as the primary target, and the person with the second highest confidence level is designated as the secondary target. Only the primary and secondary targets are retained, and the remaining targets are considered irrelevant and not output. The target detection model outputs the sequence of center points of the detected interactive objects. Again, there may be cases where multiple targets are detected in a single frame. In such cases, this invention selects the interactive object with the highest confidence level for output.

[0066] The calculation of confidence involves three aspects: first, the score of the bounding box, that is, the evaluation of whether it contains the target object; second, the probability that the bounding box is classified as "human"; and third, the detection status of key points within the bounding box, among which the confidence of key points directly affects the final confidence assessment of human.

[0067] Specifically, the confidence level is calculated using the following formula:

[0068] Conf = P(object) * IoU(pred,gt)

[0069] Where Conf represents the confidence level of each target box;

[0070] P(object) is the probability that an object exists within the frame;

[0071] IoU(pred,gt) is the maximum IoU value between the predicted box and all ground truth boxes;

[0072] The confidence score for interactive objects can be calculated using the formula described above. However, for the confidence score of a person, the confidence score is first calculated using the formula above. If the confidence score exceeds a set threshold, it is determined to be a person. Then, the confidence score for keypoints is calculated using the following formula:

[0073]

[0074] Where σ is the Sigmoid function, used to normalize the probability of the existence of keypoints;

[0075] This represents the predicted value of the key points output by the network, which can be obtained as a value between 0 and 1 using the Sigmoid function;

[0076] S22, calculate the obtained coordinates and simultaneously calculate the relative distances between multiple key points and interactive objects;

[0077] Specifically, the distance is calculated using Euclidean distance, and the specific formula is as follows: All distances are relative distances to avoid errors caused by viewpoint jitter. In addition to using Euclidean distance to calculate the relative distance between keypoints and interactive objects, other distance calculation methods can also be used, such as Manhattan distance, Chebyshev distance, cosine similarity, etc.

[0078] Where (x1, y1) are the coordinates of the interactive object in the video, and (x0, y0) are the coordinate information of the key point;

[0079] S23, store the calculated distance for later use;

[0080] S3, constructs a matrix from the time sequence information of a fixed time window;

[0081] S31, the relative distances between different key points and interactive objects are different sequences, which are used as columns of the matrix;

[0082] S32, for information with a sequence length greater than the fixed time window, equal-interval sampling is used; for information with a sequence length less than the fixed time window, linear interpolation is used to meet the set requirements.

[0083] Specifically, equal-interval sampling involves selecting samples at fixed intervals while also ensuring the representativeness of the data; linear interpolation refers to the use of linear interpolation to fill in missing data when data points are sparse.

[0084] The linear interpolation formula is as follows:

[0085]

[0086] Where y is the estimated value at x;

[0087] It is the slope between two known points;

[0088] S33. Combine the data processed in step S32 to obtain a spatiotemporal matrix. The rows of the spatiotemporal matrix represent time frames, and the columns of the spatiotemporal matrix represent the distances between different key points and interactive objects at different time frames. The data format of the spatiotemporal matrix is ​​an m-row × n-column matrix, and the values ​​of the matrix are shown in the table below.

[0089]

[0090] Here, rows represent time frames, and columns represent the distances of different key points to the interactive object at different time frames;

[0091] S34, finally the data is processed into a sorted order based on the relative distance between key points and interactive objects;

[0092] S4, use template matching algorithm to classify time series information;

[0093] S41, Based on standard actions, generate a timing template corresponding to each action. The timing template is in matrix form, where rows represent time frames and columns represent the distances of different key points to the interactive object in different time frames. These templates contain spatiotemporal trajectory information of human key points and interactive objects.

[0094] S42, the temporal template and the spatiotemporal matrix are convolved through the same filter to obtain the feature signals of the temporal template and the spatiotemporal matrix;

[0095] S43, Calculate the norm of two characteristic signals;

[0096] The Euclidean norm (L2 norm) for a vector v = [v1, v2, ..., v3] is defined as follows: n-1 ,v n Defined as:

[0097]

[0098] In terms of norm selection, the L2 norm is more suitable for current signal processing tasks than the L1 norm because the Euclidean norm has better smoothness.

[0099] S44, normalize the two characteristic signals after the norm calculation, and then calculate the cross-correlation between the two signals;

[0100] Specifically, the normalized feature signals of x(t) and y(t) are the template signal and the target signal, respectively;

[0101] Specifically, in this example, the normalization method used is the Min-Max normalization method, which normalizes the data range to 0-1;

[0102] The following formula is used to calculate cross-correlation:

[0103]

[0104] Where r(i) represents the cross-correlation of the signals, and the larger r(i) is, the greater the matching degree of the signals;

[0105] Let t represent the time delay between signals x(i) and y(i), and i be the time index of the signal. Traversing the samples of x(i) and y(i), for each value of t, i starts from the beginning index of the signal and continues until the end position of the signal.

[0106] The process of calculating cross-correlation is as follows Figure 2 As shown: x(i) represents the template signal (blue curve), y(i) represents the target signal (green curve). By sliding and comparing with the template signal, the value of t is changed to achieve the translation of the signal. In the cross-correlation calculation, the calculation process corresponds to the iteration of t in the formula. When t = 0, the starting points of the template signal and the target signal are aligned. Subsequently, as t increases, the target signal gradually shifts to the right to match other parts of the template signal.

[0107] The red curve represents the change in matching degree. The horizontal axis represents the offset t, and the vertical axis represents the cumulative matching degree (i.e., cross-correlation value). When the curve reaches its peak, it means that the template signal and the target signal have reached the best alignment state at a certain offset t, and the matching degree is the highest.

[0108] Assuming the signal length is n, the maximum value of t cannot exceed n-1. For actions with a long duration, the value of t should be larger, and vice versa.

[0109] S5, classify behaviors and actions based on similarity;

[0110] This invention provides a behavior recognition method based on the spatiotemporal matching degree of key points between people and objects, in order to solve the problem of low behavior recognition accuracy caused by ignoring the correlation between interactive objects and human behavior in the prior art.

[0111] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A behavior recognition method based on the spatiotemporal matching degree of key points between people and objects, characterized in that, Includes the following steps: S1, Preprocess the input video; S2 converts the spatiotemporal information of human body movements into the temporal information of human body key points and interactive objects; Step S2 further includes: If multiple human bodies are detected, the person with the highest confidence level is selected as the target. If multiple interactive objects are detected for the target, the interactive object with the highest confidence level is selected. The maximum confidence level includes: For the confidence level of a person, the confidence level is first calculated using formula (1). If the confidence level exceeds the set threshold, the person is identified as a person. Then, the confidence level of the key point is calculated using formula (2), and the maximum confidence level is selected from the results. For interactive objects, the confidence level is calculated directly using formula (1), and the highest confidence level is selected from the results. Formula (1) is as follows: ， in, This represents the confidence level of each target box. It predicts the probability that an object exists within the frame; It is the maximum IoU value between the predicted bounding box and all ground truth bounding boxes; Formula (2) is as follows: ， in, It is the Sigmoid function, used to normalize the probability of the existence of key points; These are key point prediction values; S3, constructs a spatiotemporal matrix from the temporal information of a fixed time window; S4, calculate the similarity between the spatiotemporal matrix and the time series template; S5. Select the time sequence template with the highest similarity. The action corresponding to the time sequence template is the behavior action of the spatiotemporal matrix.

2. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 1, characterized in that, The specific steps of step S1 are as follows: S11 processes the input video into a single-frame image using OpenCV. S12 performs noise reduction and filtering on the image processed by S11; S13: The preprocessed image is center-cropped and resized to the required dimensions to standardize the image size.

3. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 1, characterized in that, The specific steps of step S2 are as follows: S21, obtain the coordinates of various key points of the human body and the coordinates of interactive objects from the preprocessed video; the coordinates of the human body key points are obtained through the YOLO-Pose model, and the coordinates of the interactive objects are obtained through the YOLO object detection model. S22, calculate the relative distance between each key point and the interactive object.

4. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 1, characterized in that, The specific steps of step S3 are as follows: S31, the relative distances between different key points and interactive objects are different sequences, which are used as columns of the matrix; S32, for information with a sequence length greater than the fixed time window, equal-interval sampling is used, and for information with a sequence length less than the fixed time window, linear interpolation is used to meet the set time window length requirement; S33. Combine the data processed in step S32 to obtain a spatiotemporal matrix, where the rows of the spatiotemporal matrix represent time frames and the columns of the spatiotemporal matrix represent the distances of different key points to the interactive object in different time frames.

5. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41, Based on standard actions, generate a timing template corresponding to each action. The timing template is in matrix form, where rows represent time frames and columns represent the distances of different key points to the interactive object in different time frames. S42, the temporal template and the spatiotemporal matrix are convolved through the same filter to obtain the feature signals of the temporal template and the spatiotemporal matrix; S43, Calculate the norm of two characteristic signals; S44, normalize the two characteristic signals after norm calculation, and then calculate the cross-correlation between the two signals.

6. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 5, characterized in that, The formula for calculating cross-correlation is: ， in, Indicates the cross-correlation of signals; The signal length; Indicates the signal position is At that time, the value of the normalized timing template signal; Indicates the signal position is At that time, the value of the normalized spatiotemporal matrix signal; Indicates signal and The time delay between them.

7. The behavior recognition method based on the spatiotemporal matching degree of key points of people and objects according to claim 1, characterized in that, The key points of the human body are the joints of the human body.

Images