Human activity identification method and system based on comparative learning algorithm

Abstract

The invention provides a human activity identification method and system based on a comparative learning algorithm. The method comprises the following steps: setting network parameters; acquiring behavior data of the measured object, wherein the behavior data comprises image data and millimeter wave radar point cloud data; inputting the behavior data of the detected object into a human activity identification network for detection, and outputting a classification category corresponding to the behavior data by the human activity identification network; and carrying out visual display on the classification category. According to the human activity identification method and system based on the comparative learning algorithm provided by the invention, based on a comparative learning framework, the image and millimeter wave radar point cloud multi-modal data are processed through the Point Transform algorithm, and accurate classification of human body actions is realized.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence, and relates to a human activity recognition method and system based on a contrast learning algorithm. BACKGROUND

[0002] Accurate human activity recognition (HAR) is the key to realizing emerging environmental perception applications that need to understand and recognize human behavior, such as monitoring disabled or elderly people living alone. Due to the popularity of cameras, vision-based methods account for a large part of HAR, which analyzes video or pictures, extracts human skeleton joint 2D coordinates, and further classifies actions according to the joint coordinates.

[0003] Recently, wireless human sensing has become a promising solution, which uses non-intrusive sensors such as laser radar, millimeter wave radar and WiFi to solve the limitations of lighting, privacy and inconvenience. These device-free sensors propagate laser or radio frequency (RF) signals, and the responses of these signals reflect human motion at different granularities: high-resolution point cloud data of LiDAR, medium-resolution point cloud data of millimeter wave radar, and low-resolution channel state information (CSI) of WiFi. These data patterns are complementary to existing camera-based or device-based solutions and can enable more privacy-preserving human sensing applications, such as homes and hospitals.

[0004] Although single-modal human recognition methods based on CNN, Transformer, LSTM, etc. are already mature, they do not perform well in terms of recognition accuracy and practical application, thereby affecting the activity recognition result. SUMMARY

[0005] To solve the above problems, the application aims to provide a human activity recognition method and system based on a contrast learning algorithm.

[0006] The application provides a human activity recognition method based on a contrast learning algorithm, which comprises the following steps: Setting network parameters, obtaining behavior data of a measured object, wherein the behavior data comprises image data and millimeter wave radar point cloud data; Inputting the behavior data of the measured object into a human activity recognition network for detection, and outputting a classification category corresponding to the behavior data from the human activity recognition network; Visualizing and displaying the classification category.

[0007] Preferably, the camera parameters comprise: resolution: 3840x2160 @ 60fps; field of view: horizontal 80°~110°, vertical 45°~60°; focal length: 2.8~12mm motorized zoom lens; installation specification: 1.8~2.5m height, -15°±5° pitch angle, 3~5m away from the center of the target area; The millimeter wave radar parameters include: working frequency band: 77~81GHz; detection accuracy: distance resolution, speed resolution, angle resolution; detection range: maximum distance 10m, speed range 0.1~5m / s; installation specification: 0.8-1.2m height, parallel to the camera optical axis ±5°, automatic gravity calibration.

[0008] Preferably, the human activity recognition network adopts a double-encoder-single-decoder structure, and specifically includes: An image encoder: ResNet-50 backbone network, removing the fully connected layer, inputting an RGB image, and outputting a feature vector; A point cloud encoder: Point Transformer network, including: an input layer: millimeter wave radar point cloud; a 4-level feature extraction layer: each layer containing a Point Transformer module; maximum pooling outputting 512-dimensional global features; a feature fusion layer: a gated attention mechanism; and a classifier: three fully connected layers.

[0009] Preferably, the human activity recognition network adopts a MoCo contrast learning framework.

[0010] Preferably, the action classification includes: basic action, posture change, daily behavior and abnormal state. The basic action includes: walking, running, jumping and falling. The posture change includes: sitting, standing and supine. The daily behavior includes: waving hands, carrying objects and opening and closing doors. The abnormal state includes: sudden falling and long-time stillness.

[0011] The application also provides a human activity recognition system based on a contrast learning algorithm, which is used for executing the human activity recognition method based on the contrast learning algorithm. A setting module is configured to set device parameters. A data acquisition module is configured to acquire behavior data of a measured object, wherein the behavior data includes image data and millimeter wave radar point cloud data. A detection module is configured to input the behavior data of the measured object into a human activity recognition network for detection, and output a classification category corresponding to the behavior data from the human activity recognition network. A display module is configured to visually display the classification category.

[0012] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the human activity recognition method based on the contrast learning algorithm when executing the computer program.

[0013] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the human activity recognition method based on the contrast learning algorithm.

[0014] The application provides a human activity recognition method and system based on a contrast learning algorithm, which is based on a MoCo contrast learning framework, processes image and millimeter wave radar point cloud multi-modal data through a Point Transformer algorithm, and realizes human action classification. The algorithm is embedded in a computer, connected with a camera, a millimeter wave radar, a switch, a display and other related devices to form a complete human activity recognition system.

[0015] The above and other objects, advantages and features of the application will become more apparent from the following detailed description of the embodiments thereof, when read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to illustrate preferred embodiments of the application, and should not be considered limiting of the scope of the application. Like reference numerals refer to like elements throughout the drawings. In the drawings: Figure 1 is a human activity recognition method flowchart based on a contrast learning algorithm of an embodiment of the application; Figure 2 is a MoCo contrast learning framework diagram of an embodiment of the application; Figure 3 is a multi-modal network architecture diagram of an embodiment of the application; Figure 4 is a human activity recognition system structure diagram based on a contrast learning algorithm of an embodiment of the application. DETAILED DESCRIPTION

[0017] The embodiments of the application are described below in conjunction with the accompanying drawings, and those skilled in the art should understand that these embodiments are only used to explain the application, not limitative.

[0018] The application provides a human activity recognition method based on a contrast learning algorithm, as shown in Figure 1 The human activity recognition method based on the contrast learning algorithm of the embodiment of the application comprises: S1, setting network parameters.

[0019] Camera parameters: 1) Resolution: 3840x2160 @ 60fps (recommended); 2) Field of view: 80°~110° horizontally, 45°~60° vertically; 3) Focal length: 2.8~12mm motorized zoom lens (supports 3~10m effective recognition distance); 4) Installation specifications: 1.8~2.5m height, -15°±5° pitch angle, 3~5m away from the center of the target area.

[0020] Millimeter wave radar parameters: 1) Operating frequency band: 77~81GHz; 2) Detection accuracy: distance resolution (≤4cm), speed resolution (≤0.1m / s), angle resolution (horizontal 3°, vertical 8°); 3) Detection range: maximum distance 10m, speed range 0.1~5m / s; 4) Installation specifications: 0.8-1.2m height, parallel to the camera optical axis ±5°, automatic gravity calibration.

[0021] S2, obtain the behavior data of the measured object, which includes image data and millimeter wave radar point cloud data.

[0022] S3, input the behavior data of the measured object into the human activity recognition network for detection, and output the classification category corresponding to the behavior data from the human activity recognition network; Adopting a double-encoder-single-decoder structure: Image encoder: ResNet-50 backbone network (remove fully connected layer), input RGB image (224x224x3), output 1024-dimensional feature vector.

[0023] Point cloud encoder: Point Transformer network, including: input layer: millimeter wave radar point cloud (Nx5, N≤1024 points, each point containing xyz coordinates + Doppler velocity + reflectivity); 4-level feature extraction layer: each layer contains PointTransformer module (vector attention mechanism); maximum pooling output 512-dimensional global feature; feature fusion layer: gated attention mechanism; classifier: three-layer fully connected layer (1024→512→C, C is the number of activity categories).

[0024] Training process: 1) Contrastive learning pre-training stage Data augmentation strategy: Image branch: random cropping + horizontal flipping + color jittering Point cloud branch: random point sampling + affine transformation + temporal shift MoCo framework running: Query sample points (point cloud) - online encoder f_q Key samples (pictures) - momentum encoder f_k Negative sample queue: maintain 65536 historical features Loss function: InfoNCE loss 2) Downstream task fine-tuning stage Freeze image encoder, only update point cloud encoder and classifier Cross-entropy loss End-to-end training: batch size 128, AdamW optimizer (initial learning rate 3e-4, cosine decay) Take "fall detection" as an example: Data acquisition: obtain millimeter wave radar point cloud, and obtain RGB image at the same time; Feature extraction: use resnet50 and point transformer to extract features respectively; Classification decision: the classifier outputs the probability distribution: [walk: 0.02, sit: 0.11, fall: 0.87] Decision rule: if the probability of falling is greater than 0.8 and lasts for 10 frames, an alarm is triggered.

[0025] S4, the classification category is visually displayed.

[0026] Action classification includes: 1) Basic actions: walking, running, jumping, falling; 2) Posture changes: sitting, standing, lying on the back; 3) Daily behaviors: waving hands, carrying objects, opening and closing doors; 4) Abnormal states: sudden falling, long-term stillness (medical monitoring scene).

[0027] The MoCo contrast learning framework initializes the double encoder through self-supervised learning, solving the problem of multi-modal data annotation scarcity. MoCo is only a learning framework, and the feature extraction still relies on the point cloud and image encoder.

[0028] Overall, it is unsupervised representation learning using momentum contrast learning method, and the structure is as shown in Figure 2 . Figure 2 It is a MoCo contrast learning framework schematic diagram of the embodiment of the application, and the MoCo contrast learning framework of the embodiment of the application comprises Encoder (encoder): the input is "query" (query), that is, the original data, and the function of the encoder is to convert the input data into feature representation.

[0029] Momentum Encoder: used to generate the encoded representation of "key". The parameter update of Momentum Encoder is done through the momentum mechanism, usually slower than the main encoder to maintain the stability of the features. Data Augmentation: augment the original data to generate "key"; augmentation can include rotation, flipping, cropping, etc. operations, the purpose is to generate different views, but keep the same semantic information.

[0030] Feature Queue: stores the feature representation output from the Momentum Encoder, which is used to calculate the contrastive loss to help the model learn to distinguish different samples.

[0031] Logits: the features output by the encoder after contrastive loss calculation, generating logits. Logits are the raw scores predicted by the model, not converted by softmax.

[0032] One-hot Target: represents the correct matching relationship, used to supervise the calculation of contrastive loss.

[0033] Contrast Loss L self Contrastive Loss: used to measure the similarity between query and key. The goal is to maximize the similarity between query and its corresponding key, while minimizing the similarity with other keys.

[0034] Update the encoder in the way of using momentum, use the method of storing memory bank this dictionary through queue, so as to obtain a large and consistent dictionary. MoCo contrastive learning sets up positive and negative samples, obtains more samples through data augmentation (color, cropping, flipping, review, rotation, adding noise, etc.), calculates the distance between sample representations through InfoNCE loss, and gets the representation.

[0035] Figure 3 is the multi-modal network architecture diagram of the embodiment of the application. It contains two kinds of data types: point cloud and RGB picture, so a multi-modal network architecture is adopted. Among them, the RGB data uses Resnet50 network, the mmwave point cloud data uses Point Transformer network, and the multi-modal processing is realized through weighted loss function. It includes two branches, that is: Point cloud branch: establish the correspondence relationship within the modal by imposing invariance on point cloud augmentation; the input of point cloud branch is point cloud data Pi, different views of point cloud are generated from random viewpoints through 2D rendering , , using point cloud feature extractor with shared weights extract features. point cloud projection head map features to feature space and compute correspondences within point cloud branch map and to a common feature representation using mean operation .

[0036] image branch: simply formulate cross-modal correspondences by introducing a contrastive loss between rendered 2D image features and point cloud prototype features. jointly train the model with the learning objectives of both branches. input is image data extract image features using image feature extractor image projection head map features to feature space compute cross-modal correspondences between image and point cloud .

[0037] when applied to downstream tasks, drop the image branch and only use the point cloud feature extractor as the backbone for downstream tasks.

[0038] network training parameters: batch_size = 128, learning rate lr = 0.00075, optimizer optimizer = adamw, learning rate decay lr_decay_rate = 0.1, contrastive learning parameters: moco_dim = 128, moco_k = 65536, moco_m = 0.999, moco_t = 0.07, train for 100 rounds to achieve optimal results.

[0039] The embodiment of the application also provides a human activity recognition system based on a contrastive learning algorithm, which is used to execute the human activity recognition method based on the contrastive learning algorithm of the above-mentioned embodiment, and the system comprises: a setting module configured to set device parameters; a data acquisition module configured to acquire behavior data of a measured object, wherein the behavior data comprises image data and millimeter wave radar point cloud data; a detection module configured to input the behavior data of the measured object into a human activity recognition network for detection, and output a classification category corresponding to the behavior data by the human activity recognition network; a display module configured to visually display the classification category.

[0040] Specifically, the trained human activity recognition (HAR) network can be embedded into a computer, and then connected with a camera, a millimeter wave radar, a switch, a display and other related devices to form a complete human activity recognition system. A complete system structure diagram is shown in Figure 4

[0041] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall fall within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and refinements without departing from the principles of the present application shall also be considered as falling within the protection scope of the present application.​

Claims

1. A human activity recognition method based on contrastive learning algorithm, characterized in that: The method comprises: Set network parameters; Acquiring behavioral data of the measured object, wherein the behavioral data includes image data and millimeter-wave radar point cloud data; Inputting the behavior data of the subject into a human activity recognition network for detection, and having the human activity recognition network output a classification category corresponding to the behavior data; The classification categories are visually displayed.

2. The method according to claim 1, characterized in that Network parameters include camera parameters and millimeter wave radar parameters; Camera parameters include: resolution: 3840×2160 @ 60fps; field of view: 80°–110° horizontal, 45°–60° vertical; focal length: 2.8–12mm motorized zoom lens; mounting specifications: 1.8–2.5m altitude, -15°±5° pitch angle, 3–5m from the center of the target area; Millimeter-wave radar parameters include: operating frequency band: 77~81GHz; detection accuracy: distance resolution, velocity resolution, and angle resolution; detection range: maximum distance 10 meters, velocity range 0.1~5m / s; installation specifications: 0.8-1.2 meters height, parallel to the camera optical axis ±5°, and automatic gravity calibration.

3. The method according to claim 1, characterized in that The human activity recognition network adopts a dual encoder-single decoder structure, specifically including: Image encoder: ResNet-50 backbone network with fully connected layers removed, input RGB image, output feature vector; Point cloud encoder: Point Transformer network, including: input layer: millimeter wave radar point cloud; 4-level feature extraction layer: each layer contains a Point Transformer module; maximum pooling output 512-dimensional global features; feature fusion layer: gated attention mechanism; classifier: three layers of fully connected layers.

4. The method according to claim 3, characterized in that The human activity recognition network adopts the MoCo contrastive learning framework.

5. The method according to claim 1, wherein Action classification includes: basic actions, posture changes, daily behaviors and abnormal states; Basic movements include: walking, running, jumping, and falling; Posture changes include: sitting, standing, and lying down; Everyday behaviors include: waving, carrying objects, and opening and closing doors; Abnormal conditions include: sudden fall and prolonged stillness.

6. A human activity recognition system based on contrastive learning algorithm, characterized in that: The system for executing the human activity recognition method based on the contrastive learning algorithm according to claim 1 comprises: The setting module is used to set device parameters, including camera parameters and millimeter wave radar parameters; A data acquisition module, configured to acquire behavior data of the measured object based on a camera and a millimeter-wave radar, wherein the behavior data includes image data and millimeter-wave radar point cloud data; A detection module, configured to input the behavior data of the detected subject into a human activity recognition network for detection, and the human activity recognition network outputs a classification category corresponding to the behavior data; The display module is used to visually display the classification categories.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the human activity recognition method based on the contrastive learning algorithm according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for human activity recognition based on a contrastive learning algorithm according to any one of claims 1 to 5 is implemented.

Images