A human posture recognition method, system, intelligent terminal and storage medium
By generating multimodal input data through instance segmentation models and depth information, and combining classification models and scene-aided features, the accuracy problem of human pose recognition under occlusion is solved, achieving higher recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI YUANKONG AUTOMATION TECH
- Filing Date
- 2025-12-09
- Publication Date
- 2026-05-12
AI Technical Summary
In existing technologies, the accuracy of human posture recognition methods is greatly reduced when human joints are occluded, leading to inaccurate recognition.
An instance segmentation model is used to generate a binary mask and depth information of the human body region. This information is then analyzed in conjunction with multimodal input data. A classification model is used for human pose recognition, and the recognition accuracy is improved by weighted fusion of the RAFT algorithm and scene-aided features.
It reduces background interference, decreases reliance on human body joints, and improves the accuracy and robustness of human posture recognition, especially in the case of occlusion, it can still accurately recognize human posture.
Smart Images
Figure CN121281099B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of human posture recognition, and in particular to a human posture recognition method, system, smart terminal and storage medium. Background Technology
[0002] Human pose recognition mainly refers to detecting and locating key nodes of the human body (such as joints, bones, etc.) from images or videos, and estimating the human body's posture and movements.
[0003] In related technologies, human pose recognition typically employs joint detection methods (such as Open Pose). Open Pose uses convolutional neural networks to simultaneously detect key points (such as joints) of the human body and associate them with limbs, thereby assigning key points to different individuals to obtain a skeleton map of the person. Classification features are extracted from the skeleton map, and a classification model is used to infer the human pose based on the classification features.
[0004] Regarding the aforementioned technologies, joint detection methods rely on the localization of human joints. When human joints are occluded, the accuracy of joint detection methods will be greatly reduced, leading to inaccurate human posture recognition, and there is still room for improvement. Summary of the Invention
[0005] To improve the accuracy of human posture recognition, this application provides a human posture recognition method, system, smart terminal, and storage medium.
[0006] Firstly, this application provides a human posture recognition method, which adopts the following technical solution:
[0007] A human pose recognition method includes:
[0008] Acquire human body detection images;
[0009] The preset instance segmentation model is controlled to segment the human detection image to generate a human region binarized mask;
[0010] The human body region image is analyzed according to a preset depth estimation algorithm to generate image depth information;
[0011] The human detection image, the human region binarization mask, and the image depth information are analyzed to generate multimodal input data;
[0012] Multimodal input data is input into a preset classification model for classification to generate human posture types;
[0013] Determine whether the human posture type meets the requirements of the preset abnormal posture type;
[0014] If it does not meet the requirements, continue to acquire human body detection images for cyclical recognition;
[0015] If the condition is met, a prompt will be given according to the preset abnormal posture prompt information.
[0016] By adopting the above technical solution, after analyzing the human detection image, the human region binarized mask, and the image depth information, multimodal input data is obtained. This enables the classification model to identify the human pose type based on the multimodal input data. On the one hand, background interference is reduced, and on the other hand, the classification model focuses on the person to be identified, thereby getting rid of the dependence on human joints and greatly reducing the impact of occlusion, thus improving the accuracy of human pose recognition.
[0017] Optionally, the steps of analyzing the human detection image, the human region binarized mask, and image depth information to generate multimodal input data include:
[0018] Feature fusion is performed on human detection images, human region binarization masks, and image depth information to generate multimodal basic data;
[0019] Multimodal baseline data is analyzed to generate continuous frame images;
[0020] The images in consecutive frames are processed according to the preset RAFT algorithm to generate a bidirectional optical flow map;
[0021] Temporal feature processing is performed on multimodal basic data and bidirectional optical flow maps to generate temporal fusion features;
[0022] Obtain scene auxiliary features;
[0023] We perform weighted fusion of temporal fusion features and scene-assisted features to generate multimodal input data.
[0024] By adopting the above technical solution, temporal feature processing is performed on multimodal basic data and bidirectional optical flow graphs to obtain temporal fusion features, thereby constructing a dual judgment basis of static features and dynamic trends, improving the recognition accuracy of similar posture types; then, scene auxiliary features and temporal fusion features are weighted and fused, thereby adjusting the judgment threshold of the corresponding posture type in the scene according to the scene, thereby improving the accuracy of human posture recognition.
[0025] Optionally, the steps of feature fusion of the human detection image, the human region binarized mask, and image depth information to generate multimodal base data include:
[0026] The three channels of the human detection image, one channel of the human region binarization mask, and one channel of the image depth information are concatenated to generate multimodal basic data.
[0027] The expression for the multimodal basic data is:
[0028] ,
[0029] in, Represents multimodal basic data. This indicates a splicing operation at the channel level. This represents a human body detection image. Represents a binary mask for the human body region. It represents the depth information of the image.
[0030] By adopting the above technical solution, the three channels of the human body detection image, one channel of the human body region binarization mask, and one channel of the image depth information are stitched together to obtain multimodal basic data, thereby improving the efficiency of determining multimodal basic data.
[0031] Optionally, the steps of feature fusion of the human detection image, the human region binarized mask, and image depth information to generate multimodal base data include:
[0032] The binarized mask of the human body region and the image depth information are merged to generate a conditional feature map;
[0033] The human detection image, conditional feature map, preset query weight matrix, and preset key weight matrix are analyzed to generate attention weights;
[0034] The human detection images, attention weights, and preset value weight matrices are analyzed to generate multimodal basic data.
[0035] By adopting the above technical solution, the human detection image, conditional feature map, query weight matrix and key weight matrix are analyzed to obtain attention weights. Then, the human detection image, attention weights and value weight matrices are analyzed to obtain multimodal basic data, thereby achieving efficient fusion of multimodal data.
[0036] Optionally, the steps for obtaining scene auxiliary features include:
[0037] The binarized mask of the human body region and the image depth information are analyzed to generate the ground region depth variance;
[0038] The human body detection images are analyzed to generate the average image brightness.
[0039] The binarized mask of the human body region, image depth information, and preset statistical range are analyzed to generate the number of deep mutation regions;
[0040] The ground area depth variance, image brightness mean, and number of depth abrupt change regions are mapped into vectors of the same dimension as the temporal fusion features to generate scene auxiliary features.
[0041] By adopting the above technical solution, scene-assisted features are obtained, which facilitates fusion with temporal fusion features. On the other hand, specific judgment thresholds are determined based on scene-assisted features, thereby improving the accuracy of human posture recognition.
[0042] Optionally, the classification model includes a backbone network and a squeeze excitation module. The step of inputting multimodal input data into a preset classification model for classification to generate human posture types includes:
[0043] Multimodal input data is fed into the backbone network for processing to generate intermediate feature maps;
[0044] The intermediate feature map is input into the squeeze excitation module for processing to generate a recalibrated feature map;
[0045] The recalibrated feature maps are input into the backbone network for classification to generate human pose types.
[0046] By adopting the above technical solution, the intermediate feature map is input into the squeeze excitation module for processing to obtain the recalibrated feature map, which enables the squeeze excitation module to adaptively emphasize important features and suppress irrelevant features. The recalibrated feature map is then input into the backbone network for classification to obtain the human posture type, thereby improving the accuracy of human posture type recognition.
[0047] Optionally, the step of inputting the intermediate feature map into the squeeze excitation module for processing to generate a recalibrated feature map includes:
[0048] Global average pooling is performed on the intermediate feature maps to generate channel-wide features;
[0049] Analyze the global features of the channels to generate channel weights;
[0050] Channel weights and intermediate feature maps are analyzed to generate recalibrated feature maps.
[0051] By adopting the above technical solution, the intermediate feature map is subjected to global average pooling to obtain the channel global features. After analyzing the channel global features, the channel weights are obtained. The intermediate feature map is then adjusted with the channel weights to obtain the recalibrated feature map, thereby improving the efficiency and accuracy of determining the recalibrated feature map.
[0052] Secondly, this application provides a human posture recognition system, which adopts the following technical solution:
[0053] A human posture recognition system, comprising:
[0054] The acquisition module is used to acquire human detection images;
[0055] A memory for storing a program of a human posture recognition method as described in any of the preceding claims;
[0056] The processor and the program in the memory can be loaded and executed by the processor to implement a human posture recognition method as described in any of the above.
[0057] By adopting the above technical solution, the processor loads and executes a program for a human posture recognition method stored in the memory, controls the acquisition module to acquire a series of data related to human posture recognition, and then analyzes the human detection image, the human region binarized mask, and the image depth information to obtain multimodal input data. This allows the classification model to identify the human posture type based on the multimodal input data. On the one hand, this reduces background interference, and on the other hand, it allows the classification model to focus on the person to be identified, thereby getting rid of dependence on human joint points, greatly reducing the impact of occlusion, and thus improving the accuracy of human posture recognition.
[0058] Thirdly, this application provides a smart terminal, which adopts the following technical solution:
[0059] A smart terminal includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any of the preceding claims for human posture recognition.
[0060] By adopting the above technical solution, and operating a smart terminal, the processor loads and executes a computer program for a human posture recognition method stored in the memory. After analyzing the human detection image, the human region binarized mask, and the image depth information, multimodal input data is obtained. This allows the classification model to identify the human posture type based on the multimodal input data. On the one hand, background interference is reduced, and on the other hand, the classification model focuses on the person to be identified, thus getting rid of dependence on human joint points, greatly reducing the impact of occlusion, and thus improving the accuracy of human posture recognition.
[0061] Fourthly, this application provides a computer storage medium capable of storing corresponding programs, which facilitates improving the accuracy of human posture recognition, and adopts the following technical solution:
[0062] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the above-described human posture recognition methods.
[0063] By adopting the above technical solution, a computer program for a human pose recognition method is stored in a computer-readable storage medium. The processor loads and executes the computer program stored in the storage medium, thereby analyzing the human detection image, the human region binarized mask, and the image depth information to obtain multimodal input data. This allows the classification model to identify the human pose type based on the multimodal input data. On the one hand, this reduces background interference, and on the other hand, it allows the classification model to focus on the person to be identified, thus getting rid of dependence on human joints and greatly reducing the impact of occlusion, thereby improving the accuracy of human pose recognition.
[0064] In summary, this application includes at least one of the following beneficial technical effects:
[0065] 1. By analyzing human detection images, human region binarized masks, and image depth information, multimodal input data is obtained, enabling the classification model to identify human pose types based on the multimodal input data. This reduces background interference and allows the classification model to focus on the person to be identified, thereby eliminating dependence on human joints and greatly reducing the impact of occlusion, thus improving the accuracy of human pose recognition.
[0066] 2. By performing temporal feature processing on multimodal basic data and bidirectional optical flow graphs, temporal fusion features are obtained, thereby constructing a dual judgment basis of static features and dynamic trends to improve the recognition accuracy of similar posture types; then, scene auxiliary features and temporal fusion features are weighted and fused to adjust the judgment threshold of the corresponding posture type in the scene according to the scene, thereby improving the accuracy of human posture recognition.
[0067] 3. By inputting the intermediate feature map into the squeeze excitation module for processing to obtain the recalibrated feature map, the squeeze excitation module can adaptively emphasize important features and suppress irrelevant features. The recalibrated feature map is then input into the backbone network for classification to obtain the human posture type, thereby improving the accuracy of human posture type recognition. Attached Figure Description
[0068] Figure 1 This is a flowchart of a human posture recognition method in an embodiment of this application.
[0069] Figure 2 This is a flowchart of the steps in this application embodiment to analyze human detection images, human region binarization masks, and image depth information to generate multimodal input data.
[0070] Figure 3 This is a flowchart of the steps in this application embodiment to perform feature fusion on human detection images, human region binarization masks, and image depth information to generate multimodal basic data.
[0071] Figure 4 This is a flowchart of the steps for obtaining scene auxiliary features in the embodiments of this application.
[0072] Figure 5 The classification model described in this application embodiment includes a backbone network and a squeeze excitation module. The flowchart shows the steps of inputting multimodal input data into a preset classification model for classification to generate human posture types.
[0073] Figure 6 This is a flowchart of the steps in this embodiment of the application to input the intermediate feature map into the extrusion excitation module for processing in order to generate a recalibrated feature map. Detailed Implementation
[0074] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figures 1 to 6 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.
[0075] Reference Figure 1 This application discloses a human posture recognition method, including the following steps:
[0076] Step S100: Obtain human body detection images.
[0077] Among them, human body detection images refer to the posture images of the person to be detected, which are captured by a camera and sent to the processing terminal.
[0078] Step S101: Control the preset instance segmentation model to segment the human detection image to generate a human region binarized mask.
[0079] In this context, the instance segmentation model refers to the model that generates pixel-level masks for target detection. This embodiment employs YOLOv11-seg, which uses a backbone network + neck network + head network structure. The backbone network is responsible for extracting multi-scale features from the input image (low-scale features correspond to details, and high-scale features correspond to semantics). Through progressive downsampling, it generates feature maps of different scales, covering the detection needs from small to large targets. The neck network is responsible for fusing features of different scales to solve the problems of "missed detection of small targets" and "blurred detection of large targets." It adopts an improved "PAN-FPN" structure: the FPN path (top-down) passes high-scale semantic features to low-scale features, enhancing the semantic information of small targets; the PAN path (bottom-up) passes low-scale detail features to high-scale features, improving the edge accuracy of large targets. A "dynamic feature weighting" mechanism is also added, dynamically adjusting the weights of features at different scales according to the target size (e.g., small targets rely more on low-scale features), further improving fusion efficiency. The head network outputs both object detection results (category, bounding box) and instance segmentation results (mask). It includes a detection head that uses a "decoupled head" to calculate classification and regression tasks separately (the classification branch predicts the category probability, and the regression branch predicts the bounding box coordinates) to reduce interference between tasks. It also includes a segmentation head that connects a "mask branch" in parallel with the detection head to generate instance-level masks by predicting the "prototype mask" and "mask coefficients".
[0080] A human region binarization mask is an image that precisely labels human regions within an image. The mask contains only black and white pixel values. White pixel values represent regions belonging to the human body, while black pixel values represent regions belonging to the background. The processing terminal loads an instance segmentation model, reads the human detection image, and calls the inference interface of the instance segmentation model, specifying that only the "human" category is being considered (filtering other targets to improve efficiency). The model outputs bounding boxes, categories, and masks. The mask data for the human body is extracted from the inference results and scaled to the same size as the input image using bilinear interpolation to ensure pixel-level alignment. A blank mask of the same size as the original image is created as a container for the human region binarization mask. The masks of all human instances are iterated through, and regions with a value of 1 (human pixels) in each mask are marked as 255 (white) in the total mask, while non-human regions remain 0 (black), thus obtaining the human region binarization mask.
[0081] Step S102: Analyze the human body region image according to the preset depth estimation algorithm to generate image depth information.
[0082] Among them, the depth estimation algorithm refers to the model that provides accurate depth estimation for an image, which can accurately mark the depth information of objects in the image. In this embodiment, Depth Anything V2 is used.
[0083] Image depth information refers to information used to describe the spatial distance relationship between each pixel in an image and the imaging device. It is usually presented in the form of a depth map, and the data dimension is completely consistent with the size of the original image. The processing terminal loads a depth estimation algorithm, reads the human body region image, normalizes and filters noise in the human body region image, and then locks the human body region. The human body region is input into the DINOv2 encoder (base version) of Depth Anything V2 to extract multi-scale features of the human body. The specific process is as follows: shallow feature extraction (layers 1-8 of Transformer): the encoder scans the human body region through a "self-attention mechanism" to capture "local detail features"; deep feature extraction (layers 9-12 of Transformer): the encoder integrates shallow features to generate "global semantic features". The DepthAnything V2 DPT decoder receives multi-scale feature maps and generates a depth map of the human body region through "feature fusion + upsampling": First, feature fusion is performed, where the decoder fuses shallow "detail features" and deep "semantic features" according to weights; then, upsampling and depth value matching are performed, where the decoder upsamples the fused feature map to match the pixels of the human body region through "transposed convolution", and assigns a "relative depth value" to each pixel (the algorithm defaults to the smaller the depth value, the closer the object is to the lens). The original depth map is then subjected to noise removal (using "bilateral filtering" to process the depth map, eliminating sporadic abnormal depth values at the edges of the human body (such as hair strands, cuffs) (such as the misjudged "depth value 2.0" at the hair strands), making the depth changes smoother) and depth calibration (converting the "relative depth value" into "approximate absolute depth" (which needs to be combined with known scene information) - it is known that the lens was about 2 meters away from the person when shooting, so the relative depth value is converted according to "relative depth × 2 meters / maximum relative depth (1.9)") to obtain the final image depth information.
[0084] Step S103: Analyze the human detection image, the human region binarization mask, and the image depth information to generate multimodal input data.
[0085] Among them, multimodal input data refers to a comprehensive feature representation that integrates multiple core information dimensions of the image, aiming to provide more comprehensive and discriminative data support for subsequent human pose recognition. It is obtained by the processing terminal through analysis and fusion of the human detection image, the human region binarized mask, and image depth information. Specific methods are described in [reference needed]. Figure 2 The steps.
[0086] Step S104: Input the multimodal input data into a preset classification model for classification to generate human posture types.
[0087] In this context, the classification model refers to a model used to identify human pose types. This embodiment uses a CNN- or Transformer-based classification model, which includes a backbone network and a squeeze excitation module. The connection between the backbone network and the SE module mainly lies in how to improve the overall model performance by enhancing the feature representation capabilities of the backbone network. The backbone network typically refers to the basic convolutional neural network structure used to extract image features, such as ResNet or VGG. These networks extract multi-level feature representations of the input data through a series of convolutional layers, pooling layers, and possibly fully connected layers. The SE module improves network performance by explicitly modeling the interdependencies between channels. It mainly consists of three steps: squeeze, excitation, and feature recalibration. After the SE module is embedded into certain convolutional blocks of the backbone network, it can directly act on the feature maps output by those convolutional blocks, thereby adjusting the importance of each channel based on global information.
[0088] Human posture types refer to human postures, such as standing, squatting, bending over, and falling. These are identified by a classification model loaded onto the processing terminal, which then inputs multimodal input data into the model for recognition. Specific methods are detailed in [reference needed]. Figure 5 The steps.
[0089] Step S105: Determine whether the human posture type meets the requirements of the preset abnormal posture type.
[0090] Among them, abnormal posture type refers to posture that may cause injury to personnel, such as falling, etc. It is obtained by the operator sorting out postures that may cause injury to personnel. The requirement of abnormal posture type is that it exists in abnormal posture type.
[0091] By processing the terminal to determine whether the human posture type exists in the abnormal posture type, it can be determined whether the person may be in a dangerous state.
[0092] Step S1051: If it does not meet the requirements, continue to acquire human body detection images for cyclic recognition.
[0093] If the processing terminal determines that the human posture type does not exist in the abnormal posture type, it indicates that the person is not in a dangerous state. Therefore, it continues to capture images of the human body to continuously monitor changes in human posture.
[0094] Step S1052: If the condition is met, a prompt will be given according to the preset abnormal posture prompt information.
[0095] If the processing terminal determines that the human posture type is among the abnormal posture types, it indicates that the person is in a dangerous state. Therefore, it will issue a warning based on the abnormal posture prompt information so that rescuers can carry out rescue in a timely manner.
[0096] Abnormal posture prompts are messages that alert people to an abnormal posture, and can be delivered through a combination of voice, text, and light.
[0097] Reference Figure 2 The steps for generating multimodal input data by analyzing human detection images, human region binarization masks, and image depth information include:
[0098] Step S200: Perform feature fusion on the human detection image, the human region binarized mask, and the image depth information to generate multimodal basic data.
[0099] The multimodal basic data refers to the unified format feature data obtained by fusing human detection images, human region binarization masks, and image depth information. This application provides two methods for fusion. Method one involves concatenating the three channels of the human detection image, one channel of the human region binarization mask, and one channel of the image depth information to generate the multimodal basic data. The expression for the multimodal basic data is as follows:
[0100] ,
[0101] in, Represents multimodal basic data. This indicates a splicing operation at the channel level. This represents a human body detection image. Represents a binary mask for the human body region. It represents the depth information of the image.
[0102] Method 1 is a non-interactive modal fusion, where the three modal data are only parallel in the channel dimension, without any feature-level operations or interactions, but a unified multimodal input format has been formed.
[0103] Method 2 uses a cross-attention mechanism to perform interactive operations on the three modal features to obtain multimodal basic data. For details, please refer to [link / reference]. Figure 3 The steps.
[0104] Step S201: Analyze the multimodal base data to generate continuous frame images.
[0105] Among them, continuous frame images refer to continuous T-frame multimodal image data in the multimodal basic data, which are used to extract temporal fusion features with time dynamic information and enhance the model's ability to understand motion patterns.
[0106] Step S202: Process the consecutive frame images according to the preset RAFT algorithm to generate a bidirectional optical flow map.
[0107] Among them, the RAFT algorithm (Recurrent All-Pairs Field Transforms) is an algorithm used to calculate bidirectional optical flow maps. The core of RAFT is to achieve high-precision optical flow estimation through three steps: feature extraction, correlation calculation, and iterative optical flow update. Its advantage lies in using a recurrent unit (GRU) to iteratively optimize the optical flow, rather than making a one-time prediction.
[0108] A bidirectional optical flow map refers to the pixel motion vectors from frame T to frame T+1. It is obtained by the processing terminal through processing consecutive frame images using the RAFT algorithm. The specific process is as follows: first, image preprocessing is performed on the consecutive frame images; then, a convolutional neural network is used to extract multi-scale features from the two frames, divided into relevant features and contextual features. Using the relevant feature maps of the two frames as a benchmark, the dot product similarity between each pixel and all pixels in the local neighborhood of the feature map of the other frame is calculated. RAFT iteratively optimizes the optical flow through recurrent GRU units, rather than predicting it all at once. Updates are performed separately for the forward and backward optical flows, including initializing the coarse optical flow (setting the initial optical flow to a zero matrix), and adjusting the resolution and downsampled features. Figure 1 The process involves upsampling (e.g., transposed convolution) to restore the coarse optical flow resolution to the original image size for subsequent warp operations. It also includes iterative optimization, performing warp operations, relevant feature sampling, and GRU iterative updates at each iteration step; followed by multi-scale refinement. First, iterations are performed at a low resolution (e.g., 1 / 8 of the original size) to obtain the coarse optical flow; then, upsampling combined with local iterations (e.g., at 1 / 4 or 1 / 2 size) gradually restores the original image resolution, correcting local fine motion (e.g., edges, small targets). Finally, color coding is used to represent the direction and magnitude of motion, forming a bidirectional optical flow map.
[0109] Step S203: Perform temporal feature processing on the multimodal basic data and bidirectional optical flow map to generate temporal fusion features.
[0110] Temporal fusion features refer to fusion features containing temporal information, obtained by the processing terminal after performing temporal feature processing on the multimodal base data and the bidirectional optical flow graph. This application provides two methods: Method 1 is static stitching, where the bidirectional optical flow graph is treated as an independent channel and directly stitched with the multimodal base data. Method 2 is dynamic modeling of inter-frame correlation, where features are extracted from both the multimodal base data and the bidirectional optical flow graph. CNNs (such as ResNet and MobileNet) are used to extract spatial features from each frame of the multimodal base data, and a lightweight CNN is used to extract motion features from the bidirectional optical flow graph. The spatial and motion features are fused to obtain a feature sequence, which is then input into a Transformer Encoder. A self-attention mechanism is used to model inter-frame correlation, and finally, the temporal fusion features are output.
[0111] Step S204: Obtain scene auxiliary features.
[0112] Scene auxiliary features refer to a class of features used to supplement and optimize the core task (pose recognition). Essentially, they are a set of information extracted from different data sources that describes key environmental attributes of the scene. The core function of these features is to provide environmental details of the scene, allowing subsequent fusion computations to more accurately understand the context of the current scene, thereby improving the performance of the core task. For specific acquisition methods, please refer to [reference needed]. Figure 4 The steps.
[0113] Step S205: Perform weighted fusion of temporal fusion features and scene auxiliary features to generate multimodal input data.
[0114] In this step, the multimodal input data is the same as that in step S103. The processing terminal determines the weights based on scene auxiliary features. For example, in a low-light scene, the weight of the temporal fusion feature is 0.6, otherwise it is 0.8. Thus, the temporal fusion feature and the scene auxiliary feature are weighted and fused according to the weights to obtain the multimodal input data.
[0115] Reference Figure 3 The steps for fusing features from human detection images, human region binarized masks, and image depth information to generate multimodal base data include:
[0116] Step S300: The binarized mask of the human body region and the image depth information are merged to generate a conditional feature map.
[0117] Among them, the conditional feature map refers to the representation of human body region information and depth information in image processing tasks. It includes mask and depth information and is obtained by the processing terminal by merging the human body region binarized mask and image depth information along the channel dimension.
[0118] Step S301: Analyze the human detection image, conditional feature map, preset query weight matrix and preset key weight matrix to generate attention weights.
[0119] The query weight matrix and key weight matrix refer to the model parameters. They are initialized with random values and are automatically learned and updated through data during model training. The query weight matrix converts the input into a query, and the key weight matrix converts the input into a key.
[0120] Attention weights refer to the degree of attention each location in the conditional feature map receives from all locations in the human detection image feature map. The processing terminal calculates the product of the conditional feature map and the query weight matrix to obtain the Query, and calculates the product of the human detection image feature map and the key weight matrix to obtain the Key. Finally, the transpose of the Query and Key matrices is calculated, divided by a scaling factor, and then normalized along the dimension of the Key to obtain the attention weights. The scaling factor is used to prevent the gradient from vanishing due to an excessively large dot product.
[0121] Step S302: Analyze the human detection image, attention weights, and preset value weight matrix to generate multimodal basic data.
[0122] The value weight matrix refers to the model parameters, which are initialized with random values and are automatically learned and updated through data during model training. The value weight matrix is used to convert the input into a value.
[0123] The multimodal base data in this step is the same as the multimodal base data in step S200. The processing terminal calculates the product of the human detection image feature map and the value weight matrix to obtain Value. Finally, the dot product of Value and attention weight is calculated to obtain the multimodal base data.
[0124] Reference Figure 4 The steps for obtaining scene auxiliary features include:
[0125] Step S400: Analyze the binarized mask of the human body region and the image depth information to generate the ground region depth variance.
[0126] Among them, the ground region depth variance refers to the depth variance of the ground region in the image depth information. The processing terminal excludes the human body from the image depth information by using the human body region binarization mask corresponding to the human body region, retaining the ground region, thereby counting the number of pixels and the corresponding depth of the ground region, and calculating the average depth. Then, the ground region depth variance is calculated according to the variance formula based on the number of pixels, the corresponding depth of pixels and the average depth.
[0127] Step S401: Analyze the human body detection image to generate the average image brightness.
[0128] The average image brightness refers to the average brightness in the human body detection image. The processing terminal determines the pixel values of the pixels in different dimensional channels of the human body detection image, sums the pixel values, and then divides them by three times the product of the image height and width to obtain the average image brightness.
[0129] Step S402: Analyze the human body region binarized mask, image depth information, and preset statistical range to generate the number of deep mutation regions.
[0130] The statistical range refers to the range of statistical mutation areas. In this embodiment, the range within 1 meter around the human body is used as an example.
[0131] The number of depth mutation regions refers to the number of regions within 1 meter around the human body that undergo depth mutation. The processing terminal maps the human body region into a binary mask in the image depth information, and then expands outward based on the statistical range to obtain the corresponding pixel depth values. The absolute value of the difference between adjacent pixel depth values is calculated, and the maximum value of adjacent pixel depth values is selected. The quotient between the absolute value and the maximum value is then calculated to obtain the depth change rate. After determining that the depth change rate is greater than the threshold set by the user, the values are accumulated to obtain the number of depth mutation regions.
[0132] Step S403: Map the ground area depth variance, image brightness mean, and number of depth abrupt change regions into vectors of the same dimension as the temporal fusion features to generate scene auxiliary features.
[0133] In this step, the scene auxiliary features are the same as those in step S204. The processing terminal inputs the ground area depth variance, the image brightness mean, and the number of depth abrupt change regions into the MLP. The hidden layer of the MLP uses a non-linear activation function to calculate the ground area depth variance, the image brightness mean, and the number of depth abrupt change regions respectively, and finally outputs a vector with the same dimension as the temporal fusion features, which is the scene auxiliary features.
[0134] Reference Figure 5 The classification model includes a backbone network and a squeeze excitation module. The steps of inputting multimodal input data into the preset classification model for classification to generate human pose types include:
[0135] Step S500: Input the multimodal input data into the backbone network for processing to generate intermediate feature maps.
[0136] The intermediate feature map refers to the multi-channel feature map obtained after passing through some convolutional blocks. It is generated by the first few layers (convolutional layers, etc.) in the backbone network after processing the multimodal input data.
[0137] Step S501: Input the intermediate feature map into the extrusion module for processing to generate a recalibrated feature map.
[0138] The recalibrated feature map refers to the new feature map after attention adjustment, obtained by the squeeze excitation module after processing the intermediate feature map. The specific method is described in [reference needed]. Figure 6 The steps.
[0139] Step S502: Input the recalibrated feature map into the backbone network for classification to generate human pose types.
[0140] In this step, the human pose type is consistent with the human pose type in step S104. After determining the recalibrated feature map, the recalibrated feature map is passed as input to the subsequent layers of the backbone network (such as the next convolutional block, pooling layer, etc.) for further processing. The last few layers of the backbone network (global average pooling + fully connected layer) map the final feature map to the class probability, thereby completing the classification and obtaining the human pose type.
[0141] Reference Figure 6 The steps for inputting the intermediate feature map into the extrusion module for processing to generate a recalibrated feature map include:
[0142] Step S600: Perform global average pooling on the intermediate feature map to generate channel global features.
[0143] Among them, the channel global feature refers to compressing the spatial dimension information of each channel into a single value to generate a global receptive field. This is obtained by the processing terminal through global average pooling of the intermediate feature map. Specifically, it involves determining the sum of pixel values at the spatial location of the Cth channel in the intermediate feature map, and then calculating the quotient of the sum of pixel values and the number of pixels in that channel to obtain the channel global feature.
[0144] Step S601: Analyze the global features of the channel to generate channel weights.
[0145] Here, channel weight refers to the weight of each channel. After determining the global features of the channels, the weight of each channel is generated through two fully connected layers (represented by weight matrices W1 and W2, respectively) and ReLU and Sigmoid activation functions. Specifically, the product of W1 and the global features of the channel is used as the input of the ReLU activation function, and the output of the ReLU activation function is multiplied by W2 as the input of the Sigmoid activation function, thereby calculating the channel weight.
[0146] Step S602: Analyze the channel weights and intermediate feature maps to generate a recalibrated feature map.
[0147] The recalibration feature map in this step is the same as the recalibration feature map in step S501. Figure 1 The result is obtained by multiplying the channel weights and intermediate feature maps calculated by the processing terminal.
[0148] Based on the same inventive concept, embodiments of this application provide a human posture recognition system, including:
[0149] The acquisition module is used to acquire human detection images and scene auxiliary features;
[0150] A memory used to store a program for a human posture recognition method;
[0151] The processor can load and execute programs in memory to implement a human posture recognition method.
[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0153] This application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as a human posture recognition method.
[0154] Computer storage media include, for example, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media that can store program code.
[0155] Based on the same inventive concept, embodiments of this application provide a smart terminal, including a memory and a processor, wherein the memory stores a computer program that can be loaded and executed by the processor to perform a human posture recognition method.
[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0157] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, each feature is only one example of a series of equivalent or similar features.
Claims
1. A method for human posture recognition, characterized in that, include: Acquire human body detection images; The preset instance segmentation model is controlled to segment the human detection image to generate a human region binarized mask; The human body region image is analyzed according to a preset depth estimation algorithm to generate image depth information; The human detection image, the human region binarization mask, and the image depth information are analyzed to generate multimodal input data; Multimodal input data is input into a preset classification model for classification to generate human posture types; Determine whether the human posture type meets the requirements of the preset abnormal posture type; If it does not meet the requirements, continue to acquire human body detection images for cyclical recognition; If the condition is met, a prompt will be given according to the preset abnormal posture prompt information; The steps for analyzing human detection images, human region binarization masks, and image depth information to generate multimodal input data include: Feature fusion is performed on human detection images, human region binarization masks, and image depth information to generate multimodal basic data; Multimodal baseline data is analyzed to generate continuous frame images; The images in consecutive frames are processed according to the preset RAFT algorithm to generate a bidirectional optical flow map; Temporal feature processing is performed on multimodal basic data and bidirectional optical flow maps to generate temporal fusion features; Obtain scene auxiliary features; We perform weighted fusion of temporal fusion features and scene-aided features to generate multimodal input data; The steps for obtaining scene auxiliary features include: The binarized mask of the human body region and the image depth information are analyzed to generate the ground region depth variance; The human body detection images are analyzed to generate the average image brightness. The binarized mask of the human body region, image depth information, and preset statistical range are analyzed to generate the number of deep mutation regions; The ground area depth variance, image brightness mean, and number of depth abrupt change regions are mapped into vectors of the same dimension as the temporal fusion features to generate scene auxiliary features.
2. The human posture recognition method according to claim 1, characterized in that, The steps for fusing features from human detection images, human region binarization masks, and image depth information to generate multimodal base data include: The three channels of the human detection image, one channel of the human region binarization mask, and one channel of the image depth information are concatenated to generate multimodal basic data. The expression for the multimodal basic data is: ， in, Represents multimodal basic data. This indicates a splicing operation at the channel level. This represents a human body detection image. Represents a binary mask for the human body region. It represents the depth information of the image.
3. The human posture recognition method according to claim 1, characterized in that, The steps for fusing features from human detection images, human region binarized masks, and image depth information to generate multimodal baseline data include: The binarized mask of the human body region and the image depth information are merged to generate a conditional feature map; The human detection image, conditional feature map, preset query weight matrix, and preset key weight matrix are analyzed to generate attention weights; The human detection images, attention weights, and preset value weight matrices are analyzed to generate multimodal basic data.
4. The human posture recognition method according to claim 1, characterized in that, The classification model includes a backbone network and a squeeze excitation module. The steps of inputting multimodal input data into the preset classification model for classification to generate human posture types include: Multimodal input data is fed into the backbone network for processing to generate intermediate feature maps; The intermediate feature map is input into the squeeze excitation module for processing to generate a recalibrated feature map; The recalibrated feature maps are input into the backbone network for classification to generate human pose types.
5. The human posture recognition method according to claim 4, characterized in that, The steps of inputting the intermediate feature map into the squeeze excitation module for processing to generate the recalibrated feature map include: Global average pooling is performed on the intermediate feature maps to generate channel-wide features; Analyze the global features of the channels to generate channel weights; Channel weights and intermediate feature maps are analyzed to generate recalibrated feature maps.
6. A human posture recognition system, characterized in that, include: The acquisition module is used to acquire human detection images; A memory for storing a program of a human posture recognition method as described in any one of claims 1 to 5; The processor and the program in the memory can be loaded and executed by the processor to implement the human posture recognition method as described in any one of claims 1 to 5.
7. A smart terminal, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 5 for human posture recognition.
8. A computer-readable storage medium, characterized in that, The computer program is stored and can be loaded by a processor and executed as described in any one of claims 1 to 5.