Human body behavior analysis method based on multi-granularity feature fusion

Video data is obtained through high-definition cameras, combined with optical flow field and spatial gradient field for spatiotemporal analysis, converted into polar coordinate representation, and feature extraction is used for Transformer, which solves the problem of multi-grained changes in human behavior characteristics and fusion of space-time analysis, and realizes efficient human body Behavior identification and classification.

CN118968620AInactive Publication Date: 2024-11-15HEFEI YAOAN TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411027886.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to uniformly describe the multi-grained changes in human behavior characteristics, it is difficult to integrate space-time analysis with multi-grained features, and the accuracy and stability of human behavior classification are insufficient.

Method used

Video data is obtained through high-definition cameras, noise is reduced using mean filtering and smoothing technology, the human body morphology is decomposed into a linear segment combination, rotation and translation uncertainty parameters are introduced, and space-time analysis is combined with optical flow field and spatial gradient field, and converted into polar coordinate representation , use Transformer for feature extraction and classification.

Benefits of technology

It realizes a unified description of human movement and appearance characteristics, improves detection accuracy and generalization capabilities, and enhances the robustness of the system and behavior recognition capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968620A_ABST
    Figure CN118968620A_ABST
Patent Text Reader

Abstract

The invention discloses a human body behavior analysis method based on multi-granularity feature fusion, and relates to the technical field of computer vision and mode recognition, and the method specifically comprises the steps: obtaining monitoring video data through a high-definition camera, and reducing the noise through the mean filtering and smoothing technology; the human body form is decomposed into a straight line segment combination, rotation and translation uncertainty parameters are introduced into a Hough space, and multi-granularity features are extracted; human body motion and appearance information is obtained by combining an optical flow field and a space gradient field with space-time analysis; the information is converted into polar coordinate representation, and feature fusion is carried out; and finally, a naive Bayes classifier is used to classify human body behaviors. When the method is used, the defect that appearance characteristics are weak is overcome by utilizing the motion information of the human body, the detection accuracy is improved, and the system not only has stronger discrimination ability, but also can better adapt to various environments and conditions by utilizing the information of time and space domains, and the generalization ability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and pattern recognition, and in particular to a human behavior analysis method based on multi-granularity feature fusion. Background Art

[0002] In modern society, people have an increasing demand for security monitoring, intelligent security and behavior analysis. Traditional human behavior analysis methods mainly rely on single features or low-level features, such as human contours, motion trajectories, etc. These methods often fail to achieve ideal results in complex environments. With the continuous development of computer vision technology and machine learning algorithms, behavior analysis methods combining multi-granularity features have gradually become a research hotspot.

[0003] The multi-granularity feature fusion method can more comprehensively and accurately describe human behavior by extracting and combining features of different granularities. Fine-grained features can capture subtle changes in human structure and posture, and are suitable for high-precision analysis scenarios; coarse-grained features are more robust to changes in rotation and translation, and are suitable for fast and stable recognition scenarios. Therefore, the multi-granularity feature fusion method has higher robustness and accuracy.

[0004] In recent years, with the rapid development of deep learning technology, especially the widespread application of convolutional neural networks (CNN) and recurrent neural networks (RNN), human behavior analysis based on video data has made significant progress. However, these methods often require a large amount of labeled data for training and are sensitive to noise and environmental changes, resulting in less than ideal performance in practical applications. In order to improve the accuracy and stability of human behavior analysis, a human behavior analysis method based on multi-granularity feature fusion is proposed. The surveillance video data is obtained through high-definition cameras, and spatiotemporal analysis and multi-granularity feature extraction are combined to achieve accurate classification of human behavior.

[0005] However, there are several problems in the application of multi-granularity feature representation methods:

[0006] (1) How to describe the multi-granularity feature changes of human features with a unified method: Human behavioral features show diversity at different granularities. Fine-grained features may include fine posture and action details, while coarse-grained features may focus on overall behavior patterns and motion trajectories. The differences in dimensions and scales of features of different granularities make it difficult to describe them uniformly using the same method. In addition, the inconsistency of features of different granularities in spatial and temporal dimensions increases the complexity of a unified description. Fine-grained features require high-resolution spatial details, while coarse-grained features need to capture long-term behavior patterns. Therefore, it is difficult to use a unified method to describe the multi-granularity feature changes of human features.

[0007] (2) How to achieve the fusion of spatiotemporal analysis and multi-granularity features: Temporal information is continuous and dynamic, while spatial features are usually static. It is difficult to synchronize these two features in the spatiotemporal dimension and maintain data consistency and integrity. At the same time, there are differences in data structure and expression between spatiotemporal information and appearance features. Temporal features focus on motion trajectory and behavioral changes, while spatial features focus on the posture and shape of the human body. It is difficult to find an effective matching strategy in the feature extraction process so that the two can be seamlessly integrated. Therefore, how to comprehensively utilize the information in the spatiotemporal domain and integrate it with the appearance features of the human body is a difficult point.

[0008] (3) How to complete the classification of human behavior features and behavior discrimination: Human behavior is extremely diverse, including static, dynamic, continuous, discrete and other types, and the same behavior may show different characteristics in different situations. In addition, human behavior is also affected by multiple factors such as individual differences and environmental factors, which makes behavior recognition extremely complicated. And most current classifier designs rely on effective feature extraction methods. However, existing feature extraction methods often cannot fully and accurately capture the essential characteristics of human behavior. Therefore, although there are a variety of classifier design methods applied to the field of human behavior recognition to complete the behavior classification task, their correct recognition rate has not been satisfactory.

[0009] Based on the above problems, the present invention develops a human behavior analysis method based on multi-granularity feature fusion.

[0010] This method effectively describes the appearance features and motion features of human body motion in a unified manner. It can use the discriminant information provided by motion to effectively detect human bodies when the target is relatively small and the appearance features are not obvious. It has stronger discrimination ability and better generalization ability. And through multi-granularity feature representation, it has rich description capabilities and can extract structural features and statistical information at the same time. This method has significant advantages in the unified description of human motion and appearance features, and provides effective and extensive application possibilities in fields such as human body detection. Summary of the invention

[0011] The present invention provides a human behavior analysis method based on multi-granularity feature fusion, aiming to improve the recognition accuracy and robustness of a video surveillance system, and solve the problems raised in the above-mentioned background technology. The method comprises the following steps: acquiring surveillance video data through a high-definition camera, and using mean filtering and smoothing technology to reduce noise; decomposing the human body morphology into a combination of straight line segments, and introducing rotation and translation uncertainty parameters in the Hough space to extract multi-granularity features; utilizing optical flow field and spatial gradient field in combination with spatiotemporal analysis to obtain human motion and appearance information; converting this information into polar coordinate representation, and performing feature fusion; finally, using a naive Bayes classifier to classify human behavior, to ensure that the system has efficient and accurate behavior recognition capabilities in various surveillance scenarios.

[0012] The present invention provides the following technical solution: a human behavior analysis method based on multi-granularity feature fusion, comprising the following steps:

[0013] (1) Obtain video data of the monitoring scene through a high-definition camera and ensure the clarity and stability of the video data to prevent noise from affecting model training;

[0014] (2) Human body data is parsed into a combination of straight line segments as primitives, the straight line is represented by Hough space, and rotation and translation uncertainties are introduced as granularity parameters. By adjusting the granularity parameters, a multi-granularity feature representation of fine granularity features and coarse granularity features is obtained;

[0015] (3) The image is converted into feature space through mapping equations, and the granularity parameters and feature parameters are controlled to extract feature descriptors of different granularities, thus realizing a unified feature extraction process for the image;

[0016] (4) Combined with spatiotemporal analysis, the optical flow field is used to extract human motion information, and the human appearance expression is obtained by calculating the spatial gradient field, and the motion features and appearance features are uniformly represented as multi-granular features;

[0017] (5) Extract the multi-granular features of motion information and appearance information separately, convert them into polar coordinate representation and concatenate them, realize the fusion of spatiotemporal analysis and multi-granular features, and form a comprehensive human behavior feature representation;

[0018] (6) Transformer is used to input the key frame posture sequence of human motion into the embedding layer, combined with position encoding to retain the temporal information, and then feature extraction and sequence modeling are performed through a multi-layer Transformer encoder, and finally the classification results are output through a linear layer.

[0019] Preferably, the specific implementation steps of step (1) are:

[0020] The surveillance scene video data is acquired through high-definition cameras. The mean filtering method and inter-frame difference method are used to smooth the image noise, improve the clarity, and remove the noise caused by camera jitter or transmission errors. This ensures the clarity and stability of the surveillance video data, reduces the interference of noise on model training, and thus improves the accuracy of the classification model.

[0021] Preferably, the specific implementation steps of step (2) are:

[0022] Taking the straight line segment as the basic unit, the complex human body shape is decomposed into a combination of a series of straight line segments; the concept of Hough space is adopted to integrate the rotation and translation uncertainties of the curve into the definition of the curve.

[0023] Preferably, the specific implementation steps of step (3) are:

[0024] For the input image, according to different granularity parameters, the formula is described as:

[0025]

[0026] Among them, f:I→X represents a mapping from the original image space I to the feature space X; β represents the granularity parameter; α represents the feature parameter used to control the mapping type; function f(.) is a function used to extract features, and the input of this function is image I; parameter α is used to specify the feature type; parameter β is used to control the granularity; F α,β is a feature descriptor of the input image.

[0027] Preferably, the specific implementation steps of step (4) are:

[0028] By using the optical flow field as the key input for extracting human motion features for spatiotemporal analysis, the optical flow field O between two consecutive frames, image I, can be calculated by comparing the two frames. f , in the optical flow field O f In the example, each pixel is represented by a two-dimensional vector where x and y are the coordinates of the pixel point, and The x and y axes represent the movement speed or displacement component of the pixel in the X and Y axes respectively. The movement direction and speed of the pixel are estimated by comparing the pixel brightness changes in two consecutive frames, thereby constructing the optical flow field of the entire image, reflecting the movement of each part of the image between consecutive frames. Therefore, the definition of the optical flow field is as follows:

[0029]

[0030] By calculating the input image I (t) Gradient field, to obtain more information about the appearance of the human body, image I(t) Each pixel in represents a grayscale value. The grayscale change of each pixel in its neighborhood, that is, the gradient, is calculated. The spatial gradient field is defined as follows:

[0031]

[0032] Preferably, the specific implementation steps of step (5) are:

[0033] First, the original spatial coordinate quaternion Convert to polar coordinates; the original coordinates are: Where: (x) and (y) are coordinate positions, (d x ) and (d y ) is the direction vector, which is converted into polar coordinates as: [x, y, θ, ρ], where θ is the angle of the direction vector and ρ is the modulus of the direction vector; different scales are selected to describe the coordinate position and direction vector, and for each scale, its statistical characteristics and relative position information are calculated, and different time windows are selected to extract time features, and for each time window, its motion trajectory and change trend are calculated; thus, multi-granular features of spatial and temporal dimensions are extracted respectively through a two-dimensional multi-granular feature representation method;

[0034] Then, the extracted spatial multi-granularity features are combined with the temporal multi-granularity features to obtain a human behavior feature representation that combines spatiotemporal analysis with multi-granularity. The extracted spatial features and temporal features are concatenated to form a unified feature vector. The spatial and temporal features are fused together using vector concatenation or feature fusion methods to improve the robustness of the feature representation and the recognition effect.

[0035] Preferably, the specific implementation steps of step (6) are:

[0036] The key frame posture sequences of human motion are represented as time series data, and these posture sequences are used as the input of the Transformer. The posture vector is converted into a high-dimensional representation through a linear embedding layer to obtain a vector corresponding to the dimension. The time information is retained by adding position encoding, which adds a unique encoding to each time step, so that the model can perceive the time order in the sequence and better utilize the time context information. The embedded sequence is then input into the multi-layer Transformer encoder. After obtaining the output of the Transformer encoder, it is mapped to the classification result through the fully connected layer.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This human behavior analysis method based on multi-granularity feature fusion improves detection accuracy by using human motion information to make up for the shortcomings of weak appearance features. By using information in the time and space domains, the system not only has stronger discrimination capabilities, but also can better adapt to various environments and conditions, and increases generalization capabilities. Through multi-granularity feature description, human data is abstracted at different levels, from deterministic description to statistical description, providing rich description capabilities.

[0039] 2. This human behavior analysis method based on multi-granularity feature fusion effectively describes the appearance and motion features of human motion in a unified manner. It can use the discriminant information provided by motion to effectively detect human bodies when the target is relatively small and the appearance features are not obvious. It has stronger discrimination ability and better generalization ability. This method has significant advantages in the unified description of human motion and appearance features, and provides effective and extensive application possibilities in fields such as human detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of the human behavior analysis method based on multi-granularity feature fusion proposed by the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention provides a human behavior analysis method based on multi-granularity feature fusion, comprising the following steps:

[0043] (1) Obtain video data of the monitoring scene through a high-definition camera, ensure the clarity and stability of the video data, and prevent noise from affecting the training of the model; the specific implementation steps are as follows:

[0044] The surveillance scene video data is acquired through high-definition cameras. The mean filtering method and inter-frame difference method are used to smooth the image noise, improve the clarity, and remove the noise caused by camera jitter or transmission errors. This ensures the clarity and stability of the surveillance video data, reduces the interference of noise on model training, and thus improves the accuracy of the classification model.

[0045] (2) Parse the human body data into a combination of straight line segments as primitives, use Hough space to represent straight lines, and introduce rotation and translation uncertainties as granularity parameters. By adjusting the granularity parameters, a multi-granularity feature representation of fine granularity features and coarse granularity features is obtained; the specific implementation steps are:

[0046] Taking the straight line segment as the basic unit, the complex human body shape is decomposed into a series of straight line segments; the concept of Hough space is adopted to integrate the rotation and translation uncertainty of the curve into the definition of the curve. This uncertainty is quantified as a granularity parameter, and the concept of generalized curve is introduced.

[0047] When the granularity parameter is small, that is, when the rotation and translation uncertainty of the curve is low, these features will show strong geometric information and can be used to describe the details of the human body structure. They are called fine-grained features and are used in scenes that require high-precision analysis. On the contrary, when the granularity parameter is large, that is, when the rotation and translation uncertainty of the curve is high, the geometric characteristics of these features will be relatively weakened, while the statistical characteristics will be more significant. This type of feature has better robustness to changes in rotation and translation, and is called coarse-grained features. This type of feature is used in scenes that require fast and stable recognition of human body structures.

[0048] (3) The image is converted into feature space through mapping equations, and the granularity parameters and feature parameters are controlled to extract feature descriptors of different granularities, thereby realizing a unified feature extraction process for the image. The specific implementation steps are as follows:

[0049] For the input image, according to different granularity parameters, the formula is described as:

[0050]

[0051] Among them, f:I→X represents a mapping from the original image space I to the feature space X; β represents the granularity parameter; α represents the feature parameter used to control the mapping type; function f(.) is a function used to extract features, and the input of this function is image I; parameter α is used to specify the feature type; parameter β is used to control the granularity; F α,β is a feature descriptor of the input image. This function illustrates that in the granularity space framework, the feature extraction process itself is unified, and the different granularities of the features can be changed by adjusting the input parameters.

[0052] (4) Combined with spatiotemporal analysis, the optical flow field is used to extract human motion information, and the human appearance expression is obtained by calculating the spatial gradient field. The motion features and appearance features are uniformly represented as multi-granular features. The specific implementation steps are as follows:

[0053] By using the optical flow field as the key input for extracting human motion features for spatiotemporal analysis, the optical flow field O between two consecutive frames, image I, can be calculated by comparing the two frames. f , in the optical flow field O f In the example, each pixel is represented by a two-dimensional vector where x and y are the coordinates of the pixel point, and The x and y axes represent the movement speed or displacement component of the pixel in the X and Y axes respectively. The movement direction and speed of the pixel are estimated by comparing the pixel brightness changes in two consecutive frames, thereby constructing the optical flow field of the entire image, reflecting the movement of each part of the image between consecutive frames. Therefore, the definition of the optical flow field is as follows:

[0054]

[0055] By calculating the input image I (t) Gradient field, to obtain more information about the appearance of the human body, image I (t) Each pixel in represents a grayscale value, and the grayscale change of each pixel in its neighborhood, that is, the gradient, is calculated. These gradient components represent the direction and magnitude of the change of the grayscale value of the pixel in its neighborhood. Therefore, its spatial gradient field is defined as follows:

[0056]

[0057] The motion characteristics of the human body are obtained through the optical flow field, and the appearance characteristics of the human body are obtained through the spatial gradient field. Combining these two features can more accurately obtain the motion of the human body between consecutive frames.

[0058] (5) Extract the multi-granular features of motion information and appearance information separately, convert them into polar coordinate representation and concatenate them, realize the fusion of spatiotemporal analysis and multi-granular features, and form a comprehensive human behavior feature representation; the specific implementation steps are:

[0059] First, the original spatial coordinate quadruple [x, y, d x ,d y ] is converted to polar coordinates; the original coordinates are: [x, y, d x ,d y ], where (x) and (y) are coordinate positions, (d x ) and (d y) is the direction vector, which is converted to polar coordinates as: [x, y, θ, ρ], where θ is the angle of the direction vector and ρ is the modulus of the direction vector; different scales (such as coarse, medium, and fine) are selected to describe the coordinate position and direction vector. For each scale, its statistical characteristics (such as mean, variance, etc.) and relative position information are calculated. Different time windows (such as short time, medium time, and long time) are selected to extract time features. For each time window, its motion trajectory and change trend are calculated; thus, multi-granular features of spatial and temporal dimensions are extracted respectively through a two-dimensional multi-granular feature representation method.

[0060] Then, the extracted spatial multi-granularity features are combined with the temporal multi-granularity features to obtain a human behavior feature representation that combines spatiotemporal analysis with multi-granularity. A unified feature vector is formed by concatenating the extracted spatial features and temporal features; the spatial and temporal features are fused together by using vector concatenation or feature fusion methods to improve the robustness of feature representation and recognition effect.

[0061] (6) Transformer is used to input the key frame posture sequence of human motion into the embedding layer, and the temporal information is retained by combining position encoding. Then, feature extraction and sequence modeling are performed through a multi-layer Transformer encoder, and finally, the classification result is output through a linear layer. The specific implementation steps are as follows:

[0062] The key frame posture sequence of human motion is represented as time series data. These posture sequences are used as the input of the Transformer. The posture vector is converted into a high-dimensional representation through a linear embedding layer to obtain a vector corresponding to the dimension. The time information is retained by adding position encoding, which adds a unique encoding to each time step, allowing the model to perceive the time order in the sequence and better utilize the time context information. The embedded sequence is then input into the multi-layer Transformer encoder. After obtaining the output of the Transformer encoder, it is mapped to the classification result through a fully connected layer.

[0063] In summary: when this human behavior analysis method based on multi-granularity feature fusion is used, it effectively describes the appearance features and motion features reflected in the human body during movement in a unified manner. It can use the discriminant information provided by the motion to perform effective human detection when the target is relatively small and the appearance features are not obvious. It has stronger discrimination ability and better generalization ability. And through multi-granularity feature representation, it has rich description capabilities and can extract structural features and statistical information at the same time. This method has significant advantages in the unified description of human motion and appearance features, and provides effective and extensive application possibilities for fields such as human detection.

[0064] The standard parts used in the present invention can all be purchased from the market, and special-shaped parts can be customized according to the description in the specification and the drawings. The specific connection methods of each part adopt conventional means such as mature bolts, rivets, welding, etc. in the prior art. Machinery, parts and equipment all adopt conventional models in the prior art and will not be described in detail here. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field. Although the embodiments of the present invention have been shown and described, it can be understood by ordinary technicians in this field that various changes, modifications, substitutions and modifications can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.

Claims

1. A human behavior analysis method based on multi-granularity feature fusion, characterized in that: The following steps are involved: (1) Obtain video data of the monitoring scene through a high-definition camera and ensure the clarity and stability of the video data to prevent noise from affecting model training; (2) Human body data is parsed into a combination of straight line segments as primitives, the straight line is represented by Hough space, and rotation and translation uncertainties are introduced as granularity parameters. By adjusting the granularity parameters, a multi-granularity feature representation of fine granularity features and coarse granularity features is obtained; (3) The image is converted into feature space through mapping equations, and the granularity parameters and feature parameters are controlled to extract feature descriptors of different granularities, thus realizing a unified feature extraction process for the image; (4) Combined with spatiotemporal analysis, the optical flow field is used to extract human motion information, and the human appearance expression is obtained by calculating the spatial gradient field, and the motion features and appearance features are uniformly represented as multi-granular features; (5) Extract the multi-granular features of motion information and appearance information separately, convert them into polar coordinate representation and concatenate them, realize the fusion of spatiotemporal analysis and multi-granular features, and form a comprehensive human behavior feature representation; (6) Transformer is used to input the key frame posture sequence of human motion into the embedding layer, combined with position encoding to retain the temporal information, and then feature extraction and sequence modeling are performed through a multi-layer Transformer encoder, and finally the classification results are output through a linear layer.

2. The human behavior analysis method based on multi-granularity feature fusion according to claim 1 is characterized in that: The specific implementation steps of step (1) are: The surveillance scene video data is acquired through high-definition cameras. The mean filtering method and inter-frame difference method are used to smooth the image noise, improve the clarity, and remove the noise caused by camera jitter or transmission errors. This ensures the clarity and stability of the surveillance video data, reduces the interference of noise on model training, and thus improves the accuracy of the classification model.

3. The human behavior analysis method based on multi-granularity feature fusion according to claim 1, characterized in that: The specific implementation steps of step (2) are: Taking the straight line segment as the basic unit, the complex human body shape is decomposed into a combination of a series of straight line segments; the concept of Hough space is adopted to integrate the rotation and translation uncertainties of the curve into the definition of the curve.

4. The human behavior analysis method based on multi-granularity feature fusion according to claim 1, characterized in that: The specific implementation steps of step (3) are: For the input image, according to different granularity parameters, the formula is described as: {F α,β F α,β =f(I;α;β),β∈ε,F α,β ∈X} Among them, f:I→X represents a mapping from the original image space I to the feature space X; β represents the granularity parameter; α represents the feature parameter used to control the mapping type; function f(.) is a function used to extract features, and the input of this function is image I; parameter α is used to specify the feature type; parameter β is used to control the granularity; F α,β is a feature descriptor of the input image.

5. The human behavior analysis method based on multi-granularity feature fusion according to claim 1 is characterized in that: The specific implementation steps of step (4) are: By using the optical flow field as the key input for extracting human motion features for spatiotemporal analysis, the optical flow field O between two consecutive frames, image I, can be calculated by comparing the two frames. f , in the optical flow field O f In the example, each pixel is represented by a two-dimensional vector where x and y are the coordinates of the pixel point, and The x and y axes represent the movement speed or displacement component of the pixel in the X and Y axes respectively. The movement direction and speed of the pixel are estimated by comparing the pixel brightness changes in two consecutive frames, thereby constructing the optical flow field of the entire image, reflecting the movement of each part of the image between consecutive frames. Therefore, the definition of the optical flow field is as follows: By calculating the input image I (t) Gradient field, to obtain more information about the appearance of the human body, image I (t) Each pixel in represents a grayscale value. The grayscale change of each pixel in its neighborhood, that is, the gradient, is calculated. The spatial gradient field is defined as follows:

6. The human behavior analysis method based on multi-granularity feature fusion according to claim 1, characterized in that: The specific implementation steps of step (5) are: First, the original spatial coordinate quaternion Convert to polar coordinates; the original coordinates are: Where: (x) and (y) are coordinate positions, (d x ) and (d y ) is the direction vector, which is converted into polar coordinates as: [x, y, θ, ρ], where θ is the angle of the direction vector and ρ is the modulus of the direction vector; different scales are selected to describe the coordinate position and direction vector, and for each scale, its statistical characteristics and relative position information are calculated, and different time windows are selected to extract time features, and for each time window, its motion trajectory and change trend are calculated; thus, multi-granular features of spatial and temporal dimensions are extracted respectively through a two-dimensional multi-granular feature representation method; Then, the extracted spatial multi-granularity features are combined with the temporal multi-granularity features to obtain a human behavior feature representation that combines spatiotemporal analysis with multi-granularity. The extracted spatial features and temporal features are concatenated to form a unified feature vector. The spatial and temporal features are fused together using vector concatenation or feature fusion methods to improve the robustness of the feature representation and the recognition effect.

7. The human behavior analysis method based on multi-granularity feature fusion according to claim 1 is characterized in that: The specific implementation steps of step (6) are: The key frame posture sequences of human motion are represented as time series data, and these posture sequences are used as the input of the Transformer. The posture vector is converted into a high-dimensional representation through a linear embedding layer to obtain a vector corresponding to the dimension. The time information is retained by adding position encoding, which adds a unique encoding to each time step, so that the model can perceive the time order in the sequence and better utilize the time context information. The embedded sequence is then input into the multi-layer Transformer encoder. After obtaining the output of the Transformer encoder, it is mapped to the classification result through the fully connected layer.

Citation Information

Patent Citations

  • Skeleton behavior recognition method and device

    CN112446253A

  • Prevention and control risk early warning method based on aerial photography big data of unmanned aerial vehicle

    CN114120077A

  • Transform and dislocation alignment strategy-based small sample action recognition method

    CN116503945A

  • Pig behavior intelligent analysis method and system based on multi-modal semantics

    CN116959101A