Human skeleton sequence extraction and training method, device and storage medium

Abstract

The present invention discloses a method, device and storage medium for extracting and training human skeleton sequences. The training method includes: obtaining a training sample set, the training sample set includes: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal with a time dimension, the label information is the three-dimensional coordinates of each node in the human skeleton sequence of the target object in the time dimension, the first thermal map is the thermal map of the millimeter-wave radar signal in the horizontal direction, and the second thermal map is the thermal map of the millimeter-wave radar signal in the vertical direction; based on the training sample set, the human skeleton sequence extraction model is trained to obtain a trained human skeleton sequence extraction model; the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module. The human skeleton sequence can be extracted based on the millimeter-wave radar signal, with good prediction efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition, and in particular to a method, device and storage medium for extracting and training human skeleton sequences. Background Art

[0002] With the development of computer vision technology, we can perceive the behavior of objects in visual scenes based on computer vision technology, thereby realizing behavior recognition. For example, human skeleton sequence extraction based on video, but this human skeleton sequence extraction based on video has the following defects: 1. The video itself is sensitive to changes in lighting and shooting angles, which will cause subsequent difficulties in recognition; 2. The amount of video data calculation is large, which requires high computing resources.

[0003] Compared with traditional video technology, millimeter-wave radar signals can overcome uncertain factors such as background changes, lighting changes, perspective changes, and changes in human appearance, and have the characteristics of not infringing user privacy, which can greatly improve the intelligence level of equipment. In related technologies, there are two types of millimeter-wave radars: high-resolution millimeter-wave radars and low-resolution millimeter-wave radars. Among them, high-resolution millimeter-wave radars obtain rich information through a large number of transceiver antennas, and their point cloud imaging can directly see the outline and detailed information of the human body, and do not have privacy protection; and high-resolution millimeter-wave radars use dozens of antennas, the equipment is large, the power consumption is high, and there is radiation hazard to the human body. It is not suitable for home scenes and is often used for airport subway security checks; low-resolution millimeter-wave radars have a small number of antennas and limited information, so they have better privacy protection, and low-resolution millimeter-wave radars are low-cost and low-power, suitable for use in home scenes. In related technologies, gesture recognition is often based on low-resolution millimeter-wave radars, which is difficult to meet the needs of behavior recognition. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, device and storage medium for extracting and training a human skeleton sequence, aiming to extract a human skeleton sequence based on the millimeter wave radar signal of a millimeter wave radar.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a training method for a human skeleton sequence extraction model, the training method comprising:

[0007] Acquire a training sample set, the training sample set comprising: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension, the label information being the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map being a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map being a thermal map of the millimeter-wave radar signal in a vertical direction;

[0008] Training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model;

[0009] Among them, the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn the local information of the millimeter wave radar signal, the global information encoding module is used to learn the global information of the millimeter wave radar signal based on the increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

[0010] The embodiment of the present invention also provides a method for extracting a human skeleton sequence, comprising:

[0011] Obtaining a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension;

[0012] The first thermogram and the second thermogram of the millimeter wave radar signal with the time dimension are input into the human skeleton sequence extraction model trained by the training method of the human skeleton sequence extraction model described in the embodiment of the present invention to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

[0013] The embodiment of the present invention further provides a training device for a human skeleton sequence extraction model, comprising:

[0014] A first acquisition module is used to acquire a training sample set, wherein the training sample set includes: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal with a time dimension, wherein the label information is the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map is a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map is a thermal map of the millimeter-wave radar signal in a vertical direction;

[0015] A training module, used for training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model;

[0016] Among them, the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn the local information of the millimeter wave radar signal, the global information encoding module is used to learn the global information of the millimeter wave radar signal based on the increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

[0017] The embodiment of the present invention also provides a human skeleton sequence extraction device, comprising:

[0018] A second acquisition module, used to acquire a first thermal map and a second thermal map of the millimeter-wave radar signal having a time dimension;

[0019] The recognition module is used to input the first thermogram and the second thermogram of the millimeter wave radar signal with a time dimension into the human skeleton sequence extraction model trained by the training device as described in the embodiment of the present invention to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

[0020] An embodiment of the present invention further provides a training device for a human skeleton sequence extraction model, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute the steps of the training method for the human skeleton sequence extraction model described in the embodiment of the present invention when running the computer program.

[0021] An embodiment of the present invention also provides a human skeleton sequence extraction device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein the processor is used to execute the steps of the human skeleton sequence extraction method described in the embodiment of the present invention when running the computer program.

[0022] An embodiment of the present invention further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any method described in the embodiment of the present invention are implemented.

[0023] The technical solution provided by the embodiment of the present invention comprises: a training sample set including: label information, a first thermogram and a second thermogram of a millimeter-wave radar signal with a time dimension, and a human skeleton sequence extraction model including: a local information encoding module, a global information encoding module and an attention pooling module. The human skeleton sequence extraction model is trained using the training sample set to obtain a trained human skeleton sequence extraction model, so that the human skeleton sequence can be extracted based on the millimeter-wave radar signal. The human skeleton sequence extraction model of the embodiment of the present invention introduces the global information encoding module and the attention pooling module, so that the degradation problem caused by network deepening that exists in the local information encoding module performing local information learning on the input millimeter-wave radar signal can be overcome. Global information learning is performed by the global information encoding module, and attention enhancement learning is performed by the attention pooling module, so that key information can be adaptively strengthened, so that the human skeleton sequence extraction model of the embodiment of the present invention has good prediction efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A schematic flow chart of a training method for a human skeleton sequence extraction model according to an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of the structure of an experimental system for extracting human skeleton sequences based on millimeter-wave radar signals in an application embodiment of the present invention;

[0026] Figure 3 Schematic diagram of the deviation between the millimeter wave radar coordinate system and the coordinate system of the Kinect device in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of each node of a human skeleton sequence recognized by a Kinect device in an embodiment of the present invention;

[0028] Figure 5 A schematic flow chart of a training method for a human skeleton sequence extraction model in an application embodiment of the present invention;

[0029] Figure 6 A schematic diagram of the structure of a human skeleton sequence extraction model in an application embodiment of the present invention;

[0030] Figure 7 A schematic diagram of the structure of a training device for a human skeleton sequence extraction model according to an embodiment of the present invention;

[0031] Figure 8 It is a structural schematic diagram of a human skeleton sequence extraction device according to an embodiment of the present invention;

[0032] Fig. 9 A schematic diagram of the structure of a training device for a human skeleton sequence extraction model according to an embodiment of the present invention;

[0033] Fig.10 Schematic diagram of the structure of a human skeleton sequence extraction device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0036] The embodiment of the present invention provides a training method for a human skeleton sequence extraction model, such as Figure 1 As shown, the training method comprises:

[0037] Step 101, obtaining a training sample set, the training sample set including: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension, the label information being the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map being a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map being a thermal map of the millimeter-wave radar signal in a vertical direction;

[0038] Step 102: training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model.

[0039] In an embodiment of the present invention, the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn local information of the millimeter wave radar signal, the global information encoding module is used to learn global information of the millimeter wave radar signal based on an increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

[0040] In this way, the human skeleton sequence extraction model of the embodiment of the present invention constitutes a multi-angle convolutional network. Compared with the ordinary local information encoding module that learns local information of the input point cloud data stream, there is a degradation problem caused by the deepening of the network. The global information encoding module is used to learn global information, and the attention pooling module is used to perform attention enhancement learning. It can adaptively strengthen key information, so that the human skeleton sequence extraction model of the embodiment of the present invention has good prediction efficiency and accuracy.

[0041] In some embodiments, obtaining a training sample set includes: obtaining label information of the training sample set. For example, the label information in the training sample set can be obtained using a Kinect device. The Kinect device can output the three-dimensional coordinates of 25 nodes in the human skeleton sequence, and the obtained three-dimensional coordinates can be converted into a coordinate system with the millimeter-wave radar as the origin through geometric coordinate alignment, which can be used as the label information of the millimeter-wave radar signal. It can be understood that the millimeter-wave radar signal and the label information are generated for the same target object and have the same time dimension. The label information of the training sample set can be generated in other ways. For example, the three-dimensional coordinates of each node of the human skeleton sequence are generated based on the millimeter-wave radar signal. As long as it can correspond to the millimeter-wave radar signal with a time dimension, the embodiment of the present invention does not specifically limit the method of generating label information.

[0042] In some embodiments, obtaining a training sample set includes: obtaining a millimeter-wave radar signal with a time dimension, that is, a millimeter-wave radar signal data stream (for example, with a spatial three-dimensional + time dimension), and converting the millimeter-wave radar signal data stream into two three-dimensional space-time sequences (one is a two-dimensional horizontal heat map + time dimension, and the other is a two-dimensional vertical heat map + time dimension). In other examples, two two-dimensional millimeter-wave radars can be used to generate millimeter-wave radar signals in the horizontal direction and the vertical direction, respectively, that is, the first heat map and the second heat map in the embodiment of the present invention can be generated by converting the millimeter-wave radar signal collected by a three-dimensional millimeter-wave radar, or by converting the millimeter-wave radar signal collected by two two-dimensional millimeter-wave radars, and the embodiment of the present invention does not specifically limit this. In the embodiment of the present invention, the heat map of the millimeter-wave radar signal in the horizontal direction is called the first heat map, and the heat map of the millimeter-wave radar signal in the vertical direction is called the second heat map. The three-dimensional space-time sequence corresponding to the first heat map is called the first three-dimensional space-time sequence, and the three-dimensional space-time sequence corresponding to the second heat map is called the second three-dimensional space-time sequence.

[0043] In some embodiments, the training of the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model includes:

[0044] respectively performing information encoding on a first heat map and a second heat map of the millimeter-wave radar signal with a time dimension in the training sample set based on the local information encoding module, the global information encoding module and the attention pooling module;

[0045] Decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence;

[0046] Determine the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information;

[0047] Based on the network loss value, a back propagation algorithm is used to adjust the model parameters of the human skeleton sequence extraction model;

[0048] Based on the adjusted human skeleton sequence extraction model and the training sample set, continue to adjust the model parameters;

[0049] Determine whether the number of adjustments to the model parameters of the human skeleton sequence extraction model reaches a set number, or determine whether the network loss value is less than or equal to a preset threshold, to obtain a trained human skeleton sequence extraction model.

[0050] In this way, the trained human skeleton sequence extraction model obtained can support the human skeleton sequence extraction of millimeter-wave radar signals collected by millimeter-wave radar, and has good prediction efficiency and accuracy.

[0051] In some embodiments, the first heat map and the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set are respectively encoded based on the local information encoding module, the global information encoding module and the attention pooling module, including:

[0052] The first three-dimensional spatiotemporal sequence formed by the first heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a first pooling result;

[0053] The second three-dimensional spatiotemporal sequence formed by the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a second pooling result;

[0054] Accordingly, decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence includes:

[0055] The first pooling result and the second pooling result are decoded and predicted based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence.

[0056] In some embodiments, the local information encoding module is specifically used to:

[0057] The local information of the first three-dimensional space-time sequence and the second three-dimensional space-time sequence of the input is learned explicitly. By encoding the spatial geometric information at different times, the network can better learn the spatial geometric structure from the relative position and distance information of each point and the direction of the optical flow.

[0058] Exemplarily, the local information encoding module can use a resnet block (residual network block) as a basic module.

[0059] In some embodiments, the global information encoding module is specifically used to:

[0060] Learning the connection relationship between the skeleton nodes in the millimeter-wave radar signal based on the similarity between any two points in the first three-dimensional space-time sequence, and obtaining first global coding information implying the connection relationship;

[0061] Based on the similarity between any two points in the second three-dimensional space-time sequence, the connection relationship between the skeleton nodes in the millimeter-wave radar signal is learned to obtain second global coding information that implies the connection relationship.

[0062] Here, the global information encoding module is used to learn global information based on the increased network receptive field. In the related art, since the convolution operation only focuses on the local receptive field, if you want to increase the receptive field, it is generally achieved by stacking convolution layers and pooling layers, but this will increase the amount of calculation and complexity, and cause the size of the feature map to be reduced. The fully connected layer that can learn global information will bring a large number of parameters, making optimization difficult. In the task of bone sequence extraction, considering the sparsity of the point cloud map itself, it is hoped that the network can learn the connection relationship between bones. If global information can be introduced in the early stage of learning, it can solve the problem that the convolution operation cannot see the overall situation, and bring richer information to the subsequent layers. In an embodiment of the present invention, a global information encoding module is introduced. Exemplarily, the global information encoding module is defined as follows:

[0063]

[0064] Among them, x is the input, y is the output, i is the current point, and j is the other point in the input point cloud other than the current point, that is, x i is the input of the current point, x j is the input of other points other than the current point in the input point cloud, y i is the output of the current point; the f function is a function used to calculate the similarity between two points. The f function can be defined as a variant of the Gaussian function, that is, to calculate the similarity in an embedding space. The g function is a mapping function used to map a point into a vector, which can be understood as the feature of a point.

[0065] Exemplarily, the g function may be a convolution function with a convolution kernel of 1.

[0066] For example,

[0067] m(x i )=W m x i

[0068] n(x j )=W n x j

[0069]

[0070] Among them, W m , W n are the functions used in the similarity encoding process, W m or W n It can be a linear function or a convolution function, which is not specifically limited in the embodiment of the present invention.

[0071] In some embodiments, the attention pooling module is specifically used to:

[0072] Performing attention enhancement learning based on a neighborhood point set on first local encoding information output by the local information encoding module based on the first three-dimensional spatiotemporal sequence and first global encoding information output by the global information encoding module based on the first three-dimensional spatiotemporal sequence to obtain a first pooling result;

[0073] Attention enhancement learning is performed based on a neighborhood point set on the second local encoding information output by the local information encoding module based on the second three-dimensional spatiotemporal sequence and the second global encoding information output by the global information encoding module based on the second three-dimensional spatiotemporal sequence to obtain a second pooling result.

[0074] Here, the attention pooling module is used to aggregate the features of the neighborhood point set of the unit. In the related art, the heuristic max / mean etc. are often used to implement the hard aggregation of the neighborhood points, which may cause a lot of useful information to be lost. In the embodiment of the present invention, the attention pooling module automatically learns and aggregates the useful information in the neighborhood point set through attention enhancement learning. Assume that the neighborhood point set of each window in the pooling process is defined as f i ={X1, ...Xk}, that is, points 1 to k constitute a neighborhood point set, and the attention score of each point in the neighborhood point set is performed, and the formula is as follows:

[0075] s i =a(f i ,Q)

[0076] Among them, Q is the input parameter of pooling, a function is the shared function used to calculate the attention score of each point, and s i Give each point an attention score. In this way, the attention score can be used as a soft indicator for automatically selecting key features. The final feature is the weighted sum of these neighborhood point sets, defined as follows:

[0077]

[0078] in, is the characteristic output of the neighborhood point set, ∑k (f i ,s i ) is the weighted sum of the attention scores of the neighborhood point set.

[0079] In some embodiments, determining the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information includes:

[0080] The sum of the squares of the coordinate differences between the predicted three-dimensional coordinates of each node in the human skeleton sequence and the corresponding three-dimensional coordinates in the label information is counted, and the network loss value is determined based on the sum of the squares.

[0081] Exemplarily, the loss function is defined as follows:

[0082]

[0083] Among them, c is the number of bone nodes in the human skeleton sequence, x i ,y i 、z i is the coordinate value of the predicted three-dimensional coordinate of the skeleton node i, x′ i , y′ i , z′ i are the coordinate values ​​of the three-dimensional coordinates of the skeleton node i in the label information. For example, the value of c can be 17 or 25, and the embodiment of the present invention does not specifically limit the value of c.

[0084] The present invention is further described in detail below in conjunction with application examples.

[0085] like Figure 2 As shown, in this application embodiment, the experimental system for extracting human skeleton sequences based on millimeter-wave radar signals includes: a teacher network and a student network, wherein the teacher network is used to synchronously obtain the true value of the human skeleton sequence based on a Kinect device, thereby providing label information for the student network of the millimeter-wave radar signal.

[0086] During the experiment, the lens plane of the Kinect device is kept parallel to the sensor plane of the millimeter-wave radar. Among them, the millimeter-wave radar can be a common commercial millimeter-wave radar manufacturer such as TI, NXP, Continent, etc. The transmitting and receiving antennas are required to be 3-transmitter and 4-receiver or more. It is recommended to use the still object elimination algorithm built into the millimeter-wave radar system to only retain the information of active human bodies. The 3D point cloud output by the millimeter-wave radar is obtained as the point cloud to be identified, and the horizontal heat map and the vertical heat map are generated by processing. The horizontal heat map and the vertical heat map are used as the input of the student network. The Kinect device can output the 3D coordinate data of 25 nodes in the human skeleton sequence, and the 3D coordinate data is converted into the coordinate system with the millimeter-wave radar as the origin through geometric coordinate registration, which can be used as the label information of the 3D point cloud.

[0087] The above-mentioned geometric coordinate registration is exemplarily described below:

[0088] For example, it is necessary to model the geometric coordinate alignment. First, assume that the deflection angle between the coordinate system of the Kinect device and the coordinate system of the millimeter wave radar in the xyz-3 dimension is very small and can be ignored. Secondly, taking the xy plane as an example, the geometric alignment problem can be abstractly expressed as follows: Figure 3 The deviation between the mmWave radar coordinate system and the Kinect device coordinate system is shown.

[0089] like Figure 3 As shown, O(0,0) is the origin of the millimeter wave radar coordinate system, A(x' o ,y' o ) is the coordinate of the origin of the Kinect device’s coordinate system in the millimeter-wave radar coordinate system. For the unified target point M, its coordinates in the millimeter-wave radar coordinate system and the Kinect device’s coordinate system are (x 1 ,y 1 ) and (x' 1 ,y' 1 ), the coordinates of the target point in the coordinate system of the Kinect device (x' i ,y' i ) is converted into coordinates in the millimeter wave radar coordinate system

[0090]

[0091]

[0092]

[0093] Where n is the representative point in the human skeleton sequence recognized by the Kinect device.

[0094] For example, each node of the human skeleton sequence identified by the Kinect device, such as Figure 4 As shown, the embodiment of the present invention proposes 5 representative point solutions, as shown in the following Table 1.

[0095] Table 1

[0096]

[0097] Based on the corrected representative point error distance and angle, through about 6000 samples, the millimeter wave radar-signal strongest reflection point and Kinect-upper body average / not calculating the spine midpoint solution are determined. Based on such representative point selection, the geometric alignment parameters are determined. Based on the selected representative points, the parameters Δx and Δy in the above coordinate transformation formula are determined.

[0098] like Figure 5 As shown, in the application embodiment of the present invention, the training method of the human skeleton sequence extraction model includes the following steps:

[0099] Step 501, obtaining a training data set;

[0100] Obtain the 4D point cloud data (3D space + time dimension) of the spatiotemporal sequence collected by the millimeter wave radar and the label information transformed by the geometric coordinate registration as described above.

[0101] Convert 4D point cloud data into 2 3D dimension data (1 is 2D horizontal heat map + time dimension, 2 is 2D vertical heat map + time dimension). The specific conversion method is as follows:

[0102] 3D point cloud converted to 2D heat map: Based on the maximum reflection point in the millimeter wave radar, the human body position is determined, and then the starting point of the horizontal and vertical heat map and the coordinates of each node are determined with length a and b as the step length in the horizontal and vertical directions respectively. The position of the nearest point in the horizontal and vertical maps is determined based on the 3D coordinates of the point cloud image received by the millimeter wave radar in a single time, and its reflection intensity value is assigned to its nearest point.

[0103] Heat map normalization: For each graph, use the maximum value as the denominator to normalize all nodes:

[0104]

[0105] Among them, p′ i is the normalized value of the node in the heat map, p i is the current value of the node in the heat map, p max is the maximum value of the node in the heat map.

[0106] Step 502, training a human skeleton sequence extraction model based on the training sample set;

[0107] The human skeleton sequence extraction model of the embodiment of the present invention includes: a local information encoding module, a global information encoding module and an attention pooling module. The specific training process can refer to the above description and will not be repeated here.

[0108] Step 503, based on the 3D coordinates of the predicted nodes and the corresponding real coordinates, the network loss value of the skeleton sequence extraction model is calculated;

[0109] Step 504, based on the network loss value obtained from the training, the model parameters of the skeleton sequence extraction model are updated using a back propagation algorithm;

[0110] Step 505, determining whether the number of times the model parameters have been adjusted reaches the set number, or whether the network loss value is less than or equal to the preset threshold, if so, executing step 506; if not, randomly selecting a batch of training samples from the training sample set, and repeating the aforementioned steps 502 to 505;

[0111] Step 506, obtaining a trained human skeleton sequence extraction model.

[0112] The human skeleton sequence extraction model trained in the embodiment of the present invention supports the extraction of human skeleton sequences from the millimeter wave radar signal of the millimeter wave radar.

[0113] In an embodiment of the present invention, the global information encoding module solves the problems caused by the sparsity and discontinuity characteristics of low-resolution millimeter-wave point cloud signals. Considering that the low-resolution millimeter-wave point cloud signals themselves are sparse and irregular, the correspondence between points and frames is not necessarily continuous, and it is hoped that the network can learn the connection relationship between skeleton nodes. When calculating the response of a certain position, the global information encoding module of an embodiment of the present invention considers the weighting of all positions, which can be spatial or temporal, thereby effectively improving the quality of feature learning and bringing more information to the later layers.

[0114] In addition, the attention pooling module of the embodiment of the present invention reduces the loss of effective information during the network learning process. In order to ensure privacy, a low-resolution millimeter-wave radar can be used, and the information points that can be collected in each frame are limited. During the training process, in order to avoid useful information being discarded during the pooling process, the embodiment of the present invention introduces an attention pooling module to automatically learn effective information.

[0115] In this way, the human skeleton sequence extraction model of the embodiment of the present invention adopts a multi-angle convolutional network, which can efficiently extract spatial structural features with high precision requirements. Specifically, the local information encoding module is used to learn the local information of the input point cloud image stream; the global information encoding module is used to expand the convolution receptive field range, extract the global point cloud structural features, and obtain a more complete and rich representation; the attention pooling module is used to enhance the adaptive selection of important information of spatiotemporal convolution, strengthen the relationship between key point clouds in the spatiotemporal dimensions, and obtain a more robust representation, thereby improving the prediction accuracy of the entire skeleton sequence.

[0116] The embodiment of the present invention also provides a method for extracting a human skeleton sequence, comprising:

[0117] Obtaining a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension;

[0118] The first thermogram and the second thermogram of the millimeter wave radar signal with the time dimension are input into the human skeleton sequence extraction model trained by the training method of the human skeleton sequence extraction model described in the aforementioned embodiment to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

[0119] Here, obtaining the first thermal map and the second thermal map of the millimeter-wave radar signal with a time dimension may refer to the corresponding description in the aforementioned step 501, which will not be repeated here.

[0120] In an application example, the structure of the human skeleton sequence extraction model is as follows: Figure 6As shown in Figure 1, the input of the human skeleton sequence extraction model is two three-dimensional spatiotemporal sequences obtained based on millimeter-wave radar signals, namely horizontal heat map + time dimension (N*H*W*T) and vertical heat map + time dimension (N*H*W*T), where H and W are the dimensions of the heat map, T is the time dimension, and N can be understood as the number of frames. In the actual training process, it is the batch set size, wherein the horizontal point cloud image is processed by the convolution layer and output to the local information encoding module and the global information encoding module to obtain the first local encoding information and the first global encoding information, which are output to the attention pooling module after learning by resnet (residual network), and the attention pooling module outputs the first pooling result; the vertical point cloud image is processed by the convolution layer and output to the local information encoding module and the global information encoding module to obtain the second local encoding information and the second global encoding information, which are output to the attention pooling module after learning by resnet, and the attention pooling module outputs the second pooling result, and the fully connected layer connects the first pooling result and the second pooling result, and obtains the spatial three-dimensional position of each node in the human skeleton sequence through regression prediction. Exemplarily, the number of skeleton nodes can be 17 or 25 or other numbers, and the embodiment of the present invention does not specifically limit this.

[0121] For example, the label information of the training data set may come from the spatial positions of 25 skeleton nodes provided by a Kinect device, and may be processed through data preprocessing and coordinate axis matching.

[0122] In order to implement the training method of the human skeleton sequence extraction model of the embodiment of the present invention, the embodiment of the present invention also provides a training device for a human skeleton sequence extraction model. The training device for the human skeleton sequence extraction model corresponds to the training method for the human skeleton sequence extraction model mentioned above, and each step in the embodiment of the training method for the human skeleton sequence extraction model mentioned above is also fully applicable to the embodiment of the training device for the human skeleton sequence extraction model.

[0123] like Figure 7 As shown, the training device for the human skeleton sequence extraction model includes: a first acquisition module 701 and a training module 702; wherein,

[0124] The first acquisition module 701 is used to acquire a training sample set, wherein the training sample set includes: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension, wherein the label information is the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map is a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map is a thermal map of the millimeter-wave radar signal in a vertical direction;

[0125] The training module 702 is used to train the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model;

[0126] The human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn local information of the millimeter wave radar signal, the global information encoding module is used to learn global information of the millimeter wave radar signal based on an increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

[0127] In some embodiments, the training module 702 is specifically used to:

[0128] respectively performing information encoding on a first heat map and a second heat map of the millimeter-wave radar signal with a time dimension in the training sample set based on the local information encoding module, the global information encoding module and the attention pooling module;

[0129] Decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence;

[0130] Determine the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information;

[0131] Based on the network loss value, a back propagation algorithm is used to adjust the model parameters of the human skeleton sequence extraction model;

[0132] Based on the adjusted human skeleton sequence extraction model and the training sample set, continue to adjust the model parameters;

[0133] Determine whether the number of adjustments to the model parameters of the human skeleton sequence extraction model reaches a set number, or determine whether the network loss value is less than or equal to a preset threshold, to obtain a trained human skeleton sequence extraction model.

[0134] In some embodiments, the training module 702 respectively encodes the first heat map and the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set based on the local information encoding module, the global information encoding module and the attention pooling module, including:

[0135] The first three-dimensional spatiotemporal sequence formed by the first heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a first pooling result;

[0136] The second three-dimensional spatiotemporal sequence formed by the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a second pooling result;

[0137] Accordingly, decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence includes:

[0138] The first pooling result and the second pooling result are decoded and predicted based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence.

[0139] In some embodiments, the global information encoding module is specifically used to:

[0140] Learning the connection relationship between the skeleton nodes in the millimeter-wave radar signal based on the similarity between any two points in the first three-dimensional space-time sequence, and obtaining first global coding information implying the connection relationship;

[0141] Based on the similarity between any two points in the second three-dimensional space-time sequence, the connection relationship between the skeleton nodes in the millimeter-wave radar signal is learned to obtain second global coding information that implies the connection relationship.

[0142] In some embodiments, the attention pooling module is specifically used to:

[0143] Performing attention enhancement learning based on a neighborhood point set on first local encoding information output by the local information encoding module based on the first three-dimensional spatiotemporal sequence and first global encoding information output by the global information encoding module based on the first three-dimensional spatiotemporal sequence to obtain a first pooling result;

[0144] Attention enhancement learning is performed based on a neighborhood point set on the second local encoding information output by the local information encoding module based on the second three-dimensional spatiotemporal sequence and the second global encoding information output by the global information encoding module based on the second three-dimensional spatiotemporal sequence to obtain a second pooling result.

[0145] In some embodiments, the training module 702 determines the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information, including:

[0146] The sum of the squares of the coordinate differences between the predicted three-dimensional coordinates of each node in the human skeleton sequence and the corresponding three-dimensional coordinates in the label information is counted, and the network loss value is determined based on the sum of the squares.

[0147] In actual application, the first acquisition module 701 and the training module 702 can be implemented by a processor in a training device for extracting a human skeleton sequence model. Of course, the processor needs to run a computer program in a memory to implement its functions.

[0148] It should be noted that: the training device for the human skeleton sequence extraction model provided by the above embodiment only uses the division of the above program modules as an example to illustrate when training the human skeleton sequence extraction model. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the training device for the human skeleton sequence extraction model provided by the above embodiment and the training method embodiment of the human skeleton sequence extraction model belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0149] In order to implement the human skeleton sequence extraction method of the embodiment of the present invention, the embodiment of the present invention also provides a human skeleton sequence extraction device, which corresponds to the above-mentioned human skeleton sequence extraction method, and each step in the above-mentioned human skeleton sequence extraction method embodiment is also fully applicable to the present human skeleton sequence extraction device embodiment.

[0150] like Figure 8 As shown, the human skeleton sequence extraction device according to the embodiment of the present invention includes: a second acquisition module 801 and a recognition module 802; wherein,

[0151] The second acquisition module 801 is used to acquire a first thermal map and a second thermal map of the millimeter wave radar signal with a time dimension;

[0152] The identification module 802 is used to input the first thermogram and the second thermogram of the millimeter wave radar signal with the time dimension into the human skeleton sequence extraction model trained by the training device described in the embodiment of the present invention to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

[0153] In actual application, the second acquisition module 801 and the recognition module 802 can be implemented by a processor in the human skeleton sequence extraction device. Of course, the processor needs to run the computer program in the memory to implement its function.

[0154] It should be noted that: the human skeleton sequence extraction device provided by the above-described embodiment is only illustrated by the division of the above-mentioned program modules when performing human skeleton sequence extraction. In practical applications, the above-mentioned processing distribution can be completed by different program modules as needed, and the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the human skeleton sequence extraction device provided by the above-described embodiment belongs to the same concept as the human skeleton sequence extraction method embodiment. Its specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0155] Based on the hardware implementation of the above program modules, and in order to implement the training method of the human skeleton sequence extraction model in the embodiment of the present invention, the embodiment of the present invention also provides a training device for the human skeleton sequence extraction model. Fig. 9 Only the exemplary structure of the training device for the human skeleton sequence extraction model is shown, not the entire structure, and can be implemented as needed. Fig. 9 Partial or complete structure shown.

[0156] like Fig. 9 As shown, the training device 900 for extracting a human skeleton sequence model provided by an embodiment of the present invention includes: at least one processor 901, a memory 902, a user interface 903 and at least one network interface 904. The various components in the training device 900 for extracting a human skeleton sequence model are coupled together via a bus system 905. It can be understood that the bus system 905 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 905 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 905.

[0157] The user interface 903 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0158] The memory 902 in the embodiment of the present invention is used to store various types of data to support the operation of the training device for extracting a human skeleton sequence model. Examples of these data include: any computer program for operating on the training device for extracting a human skeleton sequence model.

[0159] The training method of the human skeleton sequence extraction model disclosed in the embodiment of the present invention can be applied to the processor 901, or implemented by the processor 901. The processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the training method of the human skeleton sequence extraction model can be completed by the hardware integrated logic circuit or software instructions in the processor 901. The above-mentioned processor 901 can be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 901 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiment of the present invention. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, and the storage medium is located in the memory 902. The processor 901 reads the information in the memory 902 and completes the steps of the training method of the human skeleton sequence extraction model provided by the embodiment of the present invention in combination with its hardware.

[0160] In an exemplary embodiment, the training device of the human skeleton sequence extraction model can be implemented by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), FPGA, general-purpose processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned method.

[0161] Based on the hardware implementation of the above program modules, and in order to implement the human skeleton sequence extraction method of the embodiment of the present invention, the embodiment of the present invention also provides a human skeleton sequence extraction device. Fig.10 Only the exemplary structure of the human skeleton sequence extraction device is shown, not the entire structure, and it can be implemented as needed. Fig.10 Partial or complete structure shown.

[0162] like Fig.10As shown, the human skeleton sequence extraction device 1000 provided in the embodiment of the present invention includes: at least one processor 1001, a memory 1002, a user interface 1003 and at least one network interface 1004. The various components in the human skeleton sequence extraction device 1000 are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 Various buses are labeled as bus system 1005 .

[0163] The user interface 1003 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0164] The memory 1002 in the embodiment of the present invention is used to store various types of data to support the operation of the human skeleton sequence extraction device. Examples of such data include: any computer program used to operate on the human skeleton sequence extraction device.

[0165] The human skeleton sequence extraction method disclosed in the embodiment of the present invention can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the human skeleton sequence extraction method can be completed by the hardware integrated logic circuit or software instructions in the processor 1001. The above-mentioned processor 1001 can be a general processor, a digital signal processor (DSP, DigitalSignal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiment of the present invention. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to perform, or it can be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, and the storage medium is located in the memory 1002. The processor 1001 reads the information in the memory 1002 and completes the steps of the human skeleton sequence extraction method provided by the embodiment of the present invention in combination with its hardware.

[0166] In an exemplary embodiment, the human skeleton sequence extraction device can be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, Microprocessors, or other electronic components to execute the aforementioned method.

[0167] It can be understood that the memory 902, 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0168] In an exemplary embodiment, the embodiment of the present invention further provides a storage medium, namely a computer storage medium, which can be a computer-readable storage medium, for example, including a memory 902 storing a computer program, which can be executed by a processor 901 of a training device for extracting a human skeleton sequence model to complete the steps described in the training method for extracting a human skeleton sequence model in the embodiment of the present invention; for another example, including a memory 1002 storing a computer program, which can be executed by a processor 1001 of a human skeleton sequence extraction device to complete the steps described in the human skeleton sequence extraction method in the embodiment of the present invention. The computer-readable storage medium can be a memory such as a ROM, a PROM, an EPROM, an EEPROM, a Flash Memory, a magnetic surface memory, an optical disk, or a CD-ROM.

[0169] It should be noted that: "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0170] In addition, the technical solutions described in the embodiments of the present invention can be arbitrarily combined without conflict.

[0171] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A training method for a human skeleton sequence extraction model, characterized in that: The training method comprises: Acquire a training sample set, the training sample set comprising: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension, the label information being the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map being a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map being a thermal map of the millimeter-wave radar signal in a vertical direction; Training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model; Among them, the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn the local information of the millimeter wave radar signal, the global information encoding module is used to learn the global information of the millimeter wave radar signal based on the increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

2. The method according to claim 1, characterized in that The method of training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model comprises: respectively performing information encoding on a first heat map and a second heat map of the millimeter-wave radar signal with a time dimension in the training sample set based on the local information encoding module, the global information encoding module and the attention pooling module; Decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence; Determine the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information; Based on the network loss value, a back propagation algorithm is used to adjust the model parameters of the human skeleton sequence extraction model; Based on the adjusted human skeleton sequence extraction model and the training sample set, continue to adjust the model parameters; Determine whether the number of adjustments to the model parameters of the human skeleton sequence extraction model reaches a set number, or determine whether the network loss value is less than or equal to a preset threshold, to obtain a trained human skeleton sequence extraction model.

3. The method according to claim 2, characterized in that The first heat map and the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set are respectively encoded based on the local information encoding module, the global information encoding module and the attention pooling module, including: The first three-dimensional spatiotemporal sequence formed by the first heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a first pooling result; The second three-dimensional spatiotemporal sequence formed by the second heat map of the millimeter-wave radar signal with a time dimension in the training sample set is processed by the local information encoding module and the global information encoding module respectively, and the output result of the local information encoding module and the output result of the global information encoding module are pooled based on the attention pooling module to obtain a second pooling result; Accordingly, decoding and predicting the encoding information corresponding to the first heat map and the encoding information corresponding to the second heat map based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence includes: The first pooling result and the second pooling result are decoded and predicted based on the fully connected layer to obtain the predicted three-dimensional coordinates of each node in the human skeleton sequence.

4. The method according to claim 3, characterized in that The global information encoding module is specifically used for: Learning the connection relationship between the skeleton nodes in the millimeter-wave radar signal based on the similarity between any two points in the first three-dimensional space-time sequence, and obtaining first global coding information implying the connection relationship; Based on the similarity between any two points in the second three-dimensional space-time sequence, the connection relationship between the skeleton nodes in the millimeter-wave radar signal is learned to obtain second global coding information that implies the connection relationship.

5. The method according to claim 3, characterized in that: The attention pooling module is specifically used for: Performing attention enhancement learning based on a neighborhood point set on first local encoding information output by the local information encoding module based on the first three-dimensional spatiotemporal sequence and first global encoding information output by the global information encoding module based on the first three-dimensional spatiotemporal sequence to obtain a first pooling result; Attention enhancement learning is performed based on a neighborhood point set on the second local encoding information output by the local information encoding module based on the second three-dimensional spatiotemporal sequence and the second global encoding information output by the global information encoding module based on the second three-dimensional spatiotemporal sequence to obtain a second pooling result.

6. The method according to claim 2, characterized in that The method of determining the network loss value of the human skeleton sequence extraction model based on the predicted three-dimensional coordinates of each node in the human skeleton sequence and the three-dimensional coordinates of each node in the label information includes: The sum of the squares of the coordinate differences between the predicted three-dimensional coordinates of each node in the human skeleton sequence and the corresponding three-dimensional coordinates in the label information is counted, and the network loss value is determined based on the sum of the squares.

7. A method for extracting human skeleton sequence, characterized in that: include: Obtaining a first thermal map and a second thermal map of a millimeter-wave radar signal having a time dimension; The first thermogram and the second thermogram of the millimeter wave radar signal with the time dimension are input into the human skeleton sequence extraction model trained by any one of the methods described in claims 1 to 6 to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

8. A training device for a human skeleton sequence extraction model, comprising: A first acquisition module is used to acquire a training sample set, wherein the training sample set includes: label information, a first thermal map and a second thermal map of a millimeter-wave radar signal with a time dimension, wherein the label information is the three-dimensional coordinates of each node in a human skeleton sequence of a target object in the time dimension, the first thermal map is a thermal map of the millimeter-wave radar signal in a horizontal direction, and the second thermal map is a thermal map of the millimeter-wave radar signal in a vertical direction; A training module, used for training the human skeleton sequence extraction model based on the training sample set to obtain a trained human skeleton sequence extraction model; Among them, the human skeleton sequence extraction model includes: a local information encoding module, a global information encoding module and an attention pooling module; the local information encoding module is used to learn the local information of the millimeter wave radar signal, the global information encoding module is used to learn the global information of the millimeter wave radar signal based on the increased network receptive field, and the attention pooling module is used to pool the output results of the local information encoding module and the output results of the global information encoding module.

9. A human skeleton sequence extraction device, characterized in that: include: A second acquisition module, used to acquire a first thermal map and a second thermal map of the millimeter-wave radar signal having a time dimension; The recognition module is used to input the first thermogram and the second thermogram of the millimeter wave radar signal with a time dimension into the human skeleton sequence extraction model trained by the training device as described in claim 8 to obtain the three-dimensional coordinates of each node in the human skeleton sequence.

10. A training device for a human skeleton sequence extraction model, characterized in that: include: A processor and a memory for storing a computer program that can be executed on the processor, wherein: The processor is used to execute the steps of the method according to any one of claims 1 to 6 when running a computer program.

11. A human skeleton sequence extraction device, characterized in that: include: A processor and a memory for storing a computer program that can be executed on the processor, wherein: The processor is used to execute the steps of the method according to claim 7 when running the computer program.

12. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images