A millimeter wave radar-based social intention recognition method and system

Abstract

The application discloses a social intention recognition method and system based on a millimeter wave radar. Firstly, the millimeter wave radar echo signals reflected after being modulated by human body actions in the whole target area range are acquired, and then signal processing is performed on the millimeter wave radar echo signals to remove noise and interference, so that a radio frequency tensor containing human body posture information is obtained. Then, the radio frequency tensor is input into a trained human body posture recognition model to obtain a human body target detection frame corresponding to each person in the target area range and a three-dimensional human body posture composed of human body joint points. Finally, according to the three-dimensional human body posture of each person in the target area range, the head center point position and the forward direction of each person are determined, and a classification label of whether two persons in a combination are in a social communication state is obtained by inputting a feature vector into a classification network, so that the social intention recognition is realized. The application can be used for posture recognition and social intention detection of participants in a conference or the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intention recognition, and particularly relates to a social intention recognition method and system based on a millimeter wave radar. BACKGROUND

[0002] With the increasingly common application of conference systems, the traditional conference system can only recognize the conference environment and the attendance of personnel, and cannot meet the complex requirements of detecting the actions and social intentions of the participants. The traditional human body posture and social intention recognition technology is mainly based on computer vision and wearable sensors. [1,2] The computer vision-based scheme obtains picture or video data through a camera, extracts data features for action recognition, but such a system is limited by lighting conditions, shielding and other phenomena, and changes in light and obstacles will cause a decline in recognition performance. In addition, video signals have the risk of leaking personal privacy, so they are difficult to apply in many scenarios. [3,4] The wearable sensor-based scheme has high recognition accuracy, but needs to wear special sensor equipment for each user during deployment, which is high in cost and inconvenient to use, and is difficult to popularize to most scenarios.

[0003] In this case, using a millimeter wave radar for perception and recognition has more advantages than other visual methods. The effective distance of radar detection is very far, is less affected by the outside world, and can pass through walls and other obstacles to achieve detection, meeting the recognition requirements of large and complex scenarios. Secondly, the perception mechanism of the radar can effectively avoid the possibility of privacy leakage, providing strong protection for personnel information security.

[0004] [1] Information Communication Company of State Grid Sichuan Electric Power Company, An image recognition method for a conference system: CN202010816207.1[P]. 2020-11-20.

[0005] [2] Mind Motion Technology (Shenzhen) Co., Ltd., A method for defining human head movement, posture and joint relationship through visual recognition: CN202210015299.2[P]. 2022-06-17.

[0006] [3] Tianda Technology, University of California, Human motion and position sensing, recognition and analysis based on wearable pressure sensor array: CN201880065882.6[P]. 2020-08-21.

[0007] [4] Samsung Electronics Co., Ltd., Posture recognition method, posture recognition device and wearable device: CN201610531788.8[P]. 2021-07-20. SUMMARY

[0008] The present application aims to solve the problem that it is difficult to perceive social intention by using millimeter wave radar in the prior art, and provide a millimeter wave radar-based social intention recognition method and system.

[0009] The specific technical solutions adopted by the present application are as follows:

[0010] In a first aspect, the present application provides a millimeter wave radar-based social intention recognition method, which comprises:

[0011] S1, obtaining millimeter wave radar echo signals reflected after being modulated by human body actions in the entire target area range;

[0012] S2, performing signal processing on the millimeter wave radar echo signals to remove noise and interference, thereby obtaining a radio frequency tensor containing human body posture information;

[0013] S3, inputting the radio frequency tensor into a trained human body posture recognition model to obtain a human body target detection box corresponding to each person in the target area range and a three-dimensional human body posture composed of human body joint points;

[0014] S4, determining the head center point position and the forward direction of each person according to the three-dimensional human body posture of each person in the target area range, then traversing the personnel combinations in the target area range two by two, calculating the spatial distance, the first line of sight angle and the second line of sight angle between each pair of personnel combinations and taking the three calculated information as the feature vector of the pair of personnel combinations, the spatial distance being the head center point distance between the two personnel, the first line of sight angle being the included angle between the forward direction of the first person and the head center point connecting line of the two personnel, and the second line of sight angle being the included angle between the forward direction of the second person and the head center point connecting line of the two personnel; finally, inputting the feature vector of each pair of personnel combinations into a trained classification network for binary classification to obtain the classification label of whether the two personnel in the combination are in a social communication state; based on the classification labels between the personnel two by two in the entire target area range, grouping all personnel according to the social communication relationship between each other, thereby completing social intention recognition.

[0015] As a preferred embodiment of the first aspect, the human body posture recognition model comprises a feature network, an RPN network and a pose estimation network, the radio frequency tensor is first input into the feature network to extract a feature map of the radio frequency signal, then the feature map is subjected to region of interest extraction by the RPN network; finally, the feature map extracted in the feature network and the region of interest extracted in the RPN network are input into the pose estimation network, the region of interest domain pooling layer is used to obtain the region feature map of each region of interest, then the full connection network is used for classification and regression to obtain the position, category and accurate human body target detection box of the human body joint point in each region of interest.

[0016] As a preferred form of the first aspect, the human posture recognition model is pre-trained by supervised data, and the supervised data used for training is constructed as follows:

[0017] The millimeter wave radar echo signals reflected by human actions in the whole target area range are obtained by the millimeter wave radar, and the RGB images in the whole target area range are synchronously obtained by the visible light camera, the three-dimensional skeleton information containing the joint position and category is extracted from the RGB images by the OpenPose model in advance and used as the supervised label of the millimeter wave radar echo signals.

[0018] As a preferred form of the first aspect, the classification network adopts a structural support vector machine.

[0019] As a preferred form of the first aspect, the method further comprises:

[0020] S5, the human target detection frame and the three-dimensional human posture corresponding to each person in the target area range recognized in S3 are visualized on the image, and different marks are used to distinguish the personnel information belonging to different social groups.

[0021] In a second aspect, the application provides a social intention recognition system based on a millimeter wave radar, which comprises:

[0022] A radar data acquisition module is configured to obtain millimeter wave radar echo signals reflected by human actions in the whole target area range;

[0023] A signal processing module is configured to perform signal processing on the millimeter wave radar echo signals to remove noise and interference, thereby obtaining a radio frequency tensor containing human posture information;

[0024] A posture recognition module is configured to input the radio frequency tensor into a trained human posture recognition model, thereby obtaining a human target detection frame corresponding to each person in the target area range and a three-dimensional human posture composed of human joint nodes;

[0025] The social intention recognition module is configured to determine the head center point position and the forward direction of each person according to the three-dimensional human posture of each person in the target area range, then traverse the personnel combinations in the target area range two by two, calculate the spatial distance, the first line-of-sight angle and the second line-of-sight angle between each pair of personnel combinations and take the three calculated information as the feature vector of the pair of personnel combinations, the spatial distance is the head center point distance between the two persons, the first line-of-sight angle is the included angle between the forward direction of the first person and the line connecting the head center points of the two persons, and the second line-of-sight angle is the included angle between the forward direction of the second person and the line connecting the head center points of the two persons, finally, input the feature vector of each pair of personnel combinations into the trained classification network for binary classification to obtain the classification label of whether the two persons in the combination are in the social communication state, and based on the classification labels between the two persons in the entire target area range, group all the persons according to the social communication relationship therebetween, thereby completing the social intention recognition.

[0026] As a preferred embodiment of the second aspect, the human posture recognition model comprises a feature network, an RPN network and a pose estimation network, the radio frequency tensor is first input into the feature network to extract a feature map of the radio frequency signal, then the feature map is subjected to region of interest extraction by the RPN network, finally, the feature map extracted by the feature network and the region of interest extracted by the RPN network are input into the pose estimation network, the region features of each region of interest are obtained through an interest domain pooling layer, and then the human joint point positions, categories and accurate human target detection boxes in each region of interest are obtained through a full connection network for classification and regression.

[0027] As a preferred embodiment of the second aspect, the human posture recognition model is trained in advance through supervised data, and the supervised data used for training is constructed in the following manner:

[0028] The millimeter wave radar echo signals reflected after being modulated by human actions in the entire target area range are acquired by a millimeter wave radar, and the RGB images of the entire target area range are synchronously acquired by a visible light camera, the three-dimensional skeleton information containing the joint point positions and categories is extracted from the RGB images in advance through an OpenPose model and taken as the supervised label of the millimeter wave radar echo signals.

[0029] As a preferred embodiment of the second aspect, the classification network adopts a structural support vector machine.

[0030] As a preferred embodiment of the second aspect, it further comprises:

[0031] The visualizing module is configured to visualize the human body target detection frame and the three-dimensional human body posture of each person in the target region range identified by the posture recognition module on an image, and distinguish the personnel information belonging to different social groups by different marks.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] 1) The present application can count, recognize the posture and detect the social intention of the personnel in a specific region, and can be applied to the personnel recognition scene in a conference scene, meeting more application requirements.

[0034] 2) The millimeter wave radar used in the present application can penetrate obstacles such as tables, chairs and walls, and is less affected by the environment such as light, improving the recognition accuracy and system stability, and also not leaking the privacy of the measured object.

[0035] 3) In the human body posture recognition model of the present application, the signal of each individual is amplified and separated by using the region proposal network, avoiding the estimation error caused by multipath interference or random reflection of objects in the environment, thereby improving the accuracy of posture recognition in complex scenes.

[0036] 4) The present application also designs a front-end visual processing, which not only displays the three-dimensional posture of all personnel in the measured scene, but also marks different social groups with multiple colors, so that the system recognition result is clearer. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a step schematic diagram of a social intention recognition method based on a millimeter wave radar;

[0038] Figure 2 It is a network structure diagram of a human body posture recognition model;

[0039] Figure 3 It is a module composition diagram of a social intention recognition system based on a millimeter wave radar;

[0040] Figure 4 It is a social intention recognition method flow chart in the embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below. In the following description, a lot of specific details are set forth in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0042] The social intention recognition in the present application aims to detect which people in a scene are in the same conversation group, so as to realize grouping of social groups.

[0043] As shown in the preferred embodiment of the present application, a millimeter wave radar-based social intention recognition method is provided, which comprises the following steps: Figure 1

[0044] S1, obtaining millimeter wave radar echo signals reflected after being modulated by human body actions in the entire target area range.

[0045] It should be noted that the signal acquisition here can be direct signal acquisition through the millimeter wave radar device, or indirect reading through the signals already acquired and stored by the millimeter wave radar device, which is not limited.

[0046] The acquisition of millimeter wave radar signal data belongs to the prior art, and the conventional method is to transmit wireless signals by the millimeter wave radar device, the signals are reflected after being modulated by human body actions, and the reflected signals are received by the millimeter wave receiving antenna to obtain human posture original data. The millimeter wave radar can detect signals from different three-dimensional voxels in space, and the radio frequency signals reflected from positions (x, y, z) can be calculated as:

[0047]

[0048] wherein represents the i-th sample of the frequency-modulated continuous wave (FMCW) scan received by the k-th receiving antenna at time t, λ i is the signal wavelength of the i-th sample, d k (x, y, z) represents the round-trip distance from the transmitting antenna to the voxel (x, y, z) and back to the k-th receiving antenna.

[0049] Therefore, the received millimeter wave radar echo reflection signal is a four-dimensional function of space and time, called a four-dimensional radio frequency tensor.

[0050] S2, signal processing is performed on the millimeter wave radar echo signal to remove noise and interference, so as to obtain a radio frequency tensor containing human posture information.

[0051] The purpose of the above signal processing is to remove static interference and denoise the original data, so as to obtain a multi-dimensional radio frequency tensor without interference and noise. The specific signal processing method can adopt filtering and the like, which is the same as the common preprocessing method of millimeter wave radar echo signals, and will not be described here.

[0052] ​S3, input the radio frequency tensor into a trained human posture recognition model to obtain a human body target detection frame corresponding to each person in the target region range and a three-dimensional human posture composed of human joint nodes.

[0053] The input of the human posture recognition model is a radio frequency tensor, and the output is three-dimensional human posture data of all existing persons in the detected target region range (generally composed of three-dimensional coordinate information of 14 human joint nodes such as head, neck, shoulder, elbow, wrist, hip, knee, and ankle). The neural network model for extracting the three-dimensional posture of multiple persons from the radio frequency signal includes a convolutional neural network (CNN) and a region proposal network (RPN), the CNN is used to operate the radio frequency data of each person and extract the three-dimensional skeleton thereof, and the RPN is used to detect and enlarge each human body region to separate the radio frequency data of each person, thereby extending to a multiple-person scene.

[0054] In an embodiment of the present application, as shown in Figure 2 The human posture recognition model includes a feature network, an RPN network, and a posture estimation network, the radio frequency tensor is first input into the feature network to extract a feature map of the radio frequency signal, and then the feature map is subjected to region of interest extraction by the RPN network; finally, the feature map extracted by the feature network and the region of interest extracted by the RPN network are input into the posture estimation network, the region features of each region of interest are obtained through an interest domain pooling layer, and then classification and regression are performed through a fully connected network to obtain the position, category, and accurate human target detection frame of the human joint node in each region of interest.

[0055] The feature network can generally be implemented by using a CNN convolutional network.

[0056] In addition, the human posture recognition model needs to be trained in advance before being used for actual inference, and can be used for actual inference only after the accuracy meets the requirements.

[0057] In an embodiment of the present application, in order to improve the training efficiency, the model can be trained using label samples from a visible light camera system, so that it can only predict the three-dimensional human posture according to the radio frequency signal after the training is completed. The data of the training can use a camera to shoot an RGB image synchronized with the radio frequency signal, and extract three-dimensional skeleton information from the image through a computer vision system (OpenPose), and the output here is used as a label sample for training the neural network. Therefore, in an embodiment, the human posture recognition model is trained in advance by using supervised data obtained through image processing, and the specific construction method of the supervised data used for training is as follows:

[0058] The millimeter wave radar echo signals reflected by human body actions in the whole target area range are acquired by the millimeter wave radar, and the RGB images in the whole target area range are synchronously acquired by the visible light camera, the three-dimensional skeleton information containing the joint position and category is extracted from the RGB images in advance by the OpenPose model and is taken as the supervised label of the millimeter wave radar echo signals.

[0059] S4, according to the three-dimensional human posture of each person in the target area range, the head center point position and the forward direction of each person are determined, then the personnel combinations in the target area range are traversed two by two, the spatial distance, the first line of sight angle and the second line of sight angle between each pair of personnel combinations are calculated and the three calculated information are taken as the feature vector of the pair of personnel combinations, the spatial distance is the head center point distance between two persons, the first line of sight angle is the included angle between the forward direction of the first person and the head center point connection line of two persons, and the second line of sight angle is the included angle between the forward direction of the second person and the head center point connection line of two persons; finally, the feature vector of each pair of personnel combinations is input into the trained classification network for binary classification, and the classification label of whether two persons in the combination are in the social communication state is obtained; based on the classification label between two persons in the whole target area range, all persons are socially grouped according to the social communication relationship between each other, so that the social intention recognition is completed.

[0060] In the embodiment of the application, the head center point position and the forward direction of each person can be determined according to the human joint in the three-dimensional human posture information. The head center point position can be determined according to the head joint, and the forward direction can generally adopt the forward direction parallel to the sagittal plane of the head.

[0061] In order to realize the social group grouping purpose of the social intention recognition in the application, it is necessary to classify whether all persons are in the social communication state. According to the f-formation theory, people tend to approach the interactive person and adjust the direction when interacting, so as to naturally focus on the interesting theme. Therefore, in the above S4 step, the distance and direction estimation between each individual are calculated according to the three-dimensional human posture information obtained in S3, which is used to describe the mutual relationship of a pair of persons in the scene, and whether the crowd constitutes a social group is judged according to the mutual distance and direction of the subjects in the scene.

[0062] The classification network can adopt any neural network capable of realizing the corresponding function. In the embodiments of the present application, the classification network can be realized by a structural support vector machine (Structural SVM). The grouping of social groups by the classification network is essentially a related clustering algorithm based on structural support vector machine, which can combine pairs of people into related groups. Since the direction and distance can assume different meanings in different social contexts, the structural support vector machine is used to learn how to weight the feature vectors of the related clustering according to the social context.

[0063] In the embodiments of the present application, when grouping all personnel according to the social communication relationship between each other, the classification labels between each pair of personnel in the entire target area range need to be considered, and the specific grouping principle can be adjusted according to the actual situation. Generally speaking, for a group of personnel in a social communication state, the group can be first taken as a clustering category, and then all other personnel are traversed. If a certain personnel is also in a social communication state with any personnel in the clustering category, the personnel is also added to the clustering category. In this way, all personnel are divided into multiple clustering categories, and all personnel in each clustering category form a social group, representing that the personnel have a social communication relationship.

[0064] Of course, the above grouping principle is only a preferred way, and in actual application, the time sequence association can also be considered, that is, if at a certain moment, a certain personnel in the traversal process does not have a social communication state with any personnel in a certain clustering category, but has a social communication state within a certain time period before and after, the personnel can also be included in the clustering category. Therefore, the core of the present application is to detect whether there is a social communication state between personnel, but the specific grouping principle can be adjusted according to actual needs, and no limitation is made to this.

[0065] In the embodiments of the present application, the above social intention recognition method based on millimeter wave radar further includes a visualization step, which is specifically as follows:

[0066] S5, visualizing the human body target detection frame and three-dimensional human body posture corresponding to each person in the target area range identified in S3 on an image, and distinguishing the personnel information belonging to different social groups by different markers.

[0067] The markers for distinguishing the personnel information belonging to different social groups can be color, shape, text, or even a graph structure composed of nodes and edges. It is preferred to recommend using different colors to distinguish the personnel information belonging to different social groups, for example, the human body target detection frame can be marked in different colors, and the corresponding group number can also be marked on the frame.

[0068] Based on the same inventive concept, another preferred embodiment of the present application also provides a millimeter-wave radar-based social intention recognition system corresponding to the millimeter-wave radar-based social intention recognition method provided by the above-mentioned embodiment. As shown in Figure 3 The millimeter-wave radar-based social intention recognition system includes several basic modules, which are respectively:

[0069] The radar data acquisition module is used to acquire the millimeter-wave radar echo signals reflected after being modulated by human body actions in the entire target area range.

[0070] The signal processing module is used to perform signal processing on the millimeter-wave radar echo signals to remove noise and interference, so as to obtain a radio frequency tensor containing human body posture information.

[0071] The posture recognition module is used to input the radio frequency tensor into a trained human body posture recognition model, so as to obtain a human body target detection frame corresponding to each person in the target area range and a three-dimensional human body posture composed of human body joint points.

[0072] The social intention recognition module is used to determine the head center point positions and the front view directions of each person according to the three-dimensional human body postures of each person in the target area range, then iterates through the personnel combinations in the target area range two by two, calculates the spatial distance, the first line-of-sight angle and the second line-of-sight angle between each pair of personnel combinations and takes the three calculated information as the feature vector of the pair of personnel combinations, the spatial distance is the head center point distance between the two persons, the first line-of-sight angle is the included angle between the front view direction of the first person and the line connecting the head center points of the two persons, and the second line-of-sight angle is the included angle between the front view direction of the second person and the line connecting the head center points of the two persons; finally, the feature vector of each pair of personnel combinations is input into a trained classification network for binary classification to obtain the classification label of whether the two persons in the combination are in a social communication state; based on the classification labels between the personnel two by two in the entire target area range, all personnel are socially grouped according to the social communication relationship between each other, so as to complete the social intention recognition.

[0073] In addition, the above-mentioned millimeter-wave radar-based social intention recognition system also includes a visualization module, which is used to visualize the human body target detection frame and the three-dimensional human body posture corresponding to each person in the target area range recognized by the posture recognition module on an image, and distinguish the personnel information belonging to different social groups by different markers.

[0074] Since the principle of solving problems of the millimeter-wave radar-based social intention recognition system in the embodiment of the present application is similar to that of the millimeter-wave radar-based social intention recognition method of the above-mentioned embodiment of the present application, the specific implementation forms of the modules of the system in this embodiment can also be referred to the specific implementation forms of the above-mentioned method, and the repeated parts will not be described again.

[0075] The above-mentioned method of S1-S5 of the present application will be described below by a specific embodiment to facilitate understanding.

[0076] Embodiment

[0077] In this embodiment, the above-mentioned millimeter wave radar-based social intention recognition method is applied to a conference scenario to recognize the social intention of the participants and cluster the participants in a social communication state. As shown in FIG. 1, the training and inference process of this method is implemented according to the following steps: Figure 4

[0078] First, millimeter wave radar data is collected. The data collection equipment of this embodiment uses a 60GHz millimeter wave module IWR6843ISK-ODS and an evaluation board MMWAVEICBOOST. One radar module is placed horizontally and one is placed vertically to obtain the millimeter wave radar echo signal reflected by the human body in the entire target area range. The software uses mmWave SDK and CodeComposer Studio (CCS).

[0079] Second, millimeter wave radar echo data processing is performed to remove noise and interference and obtain a four-dimensional radio frequency tensor containing distance, angle, etc. information without noise interference. The four-dimensional radio frequency tensor contains human posture information and can be used for subsequent human posture recognition.

[0080] Third, a neural network model is designed and trained for human posture recognition. The model uses a four-dimensional radio frequency tensor as input and outputs a three-dimensional human posture.

[0081] The space is discretized into three-dimensional voxels. The purpose of the designed human posture recognition model is to classify the position of each key point (head, neck, right elbow, etc.) into one voxel based on the four-dimensional radio frequency tensor. The model needs to aggregate information in space and assign a score to each spatial voxel; at the same time, the model also needs to aggregate temporal information to infer the key points that may be occluded at a specific time. Therefore, the model takes the four-dimensional radio frequency tensor as the input of the feature network, performs four-dimensional convolution at each layer, and aggregates information along space-time, i.e.:

[0082] a n =f n *a (4D) a n-1 ,

[0083] where a n and a n-1 are the feature mappings of the nth layer and the (n-1)th layer, f n is the four-dimensional convolution filter of the nth layer, and (4D) is the four-dimensional convolution operator.

[0084] For all three-dimensional voxels v e V, the model outputs the corresponding score s = {s v} v∈V , the target voxel v * is the voxel containing the joint. For the single-person pose estimation task, in order to locate the 14 joints, the CNN outputs the score s k for each joint k respectively. The total loss of pose estimation is the sum of the Softmax loss of all joints:

[0085]

[0086]

[0087] After the model training is completed, the position of the joint k can be predicted as the voxel with the highest score:

[0088]

[0089] However, in the conference scene, the separation of multiple people's radio frequency signals needs to be considered, so the CNN model is expanded by adding an RPN (Region Proposal Network, Region Proposal) network, and the structure is referred to Faster R-CNN. The final designed human pose recognition model is shown in Figure 3 After inserting the RPN, the network model is divided into two parts: the feature network, the RPN, and the pose estimation network. The input of the feature network is the radio frequency tensor, and the output is the feature map of the radio frequency signal. The feature network is composed of basic conv+relu+pooling layers, and the extracted feature map is used for the subsequent RPN network and pose estimation network. The input of the RPN network is the feature map extracted in the feature network, and the output is a plurality of regions of interest (ROI), i.e., potential human body regions. The pose estimation network is divided into a ROI pooling layer, a fully connected network, a classification layer, and a regression layer. The ROI pooling layer takes the ROI output by the RPN and the feature map output by the feature network as input. For each region detected by the RPN, the corresponding region in the feature map is enlarged and cropped to obtain a fixed-size region feature map (proposal feature map) as output. The output of the ROI pooling layer is input into the classification layer and the regression layer after passing through two fully connected layers with relu activation function. The input of the classification layer and the regression layer is the fully connected result of the region feature map of the previous layer, and the output is the class to which the joint in the region of interest belongs and the accurate target detection frame. This part calculates through the fully connected layer and the softmax which three-dimensional voxel the joint in each region belongs to, so as to obtain the three-dimensional coordinates of the joint; at the same time, the more accurate human target detection frame is obtained by using the bounding box regression.

[0090] In the above human posture recognition model, RPN is introduced to amplify and separate the signal of each individual, avoiding the estimation error caused by multipath interference or random reflection of objects in the environment, thereby improving the accuracy of posture recognition in complex scenes. The structure and principle of RPN network belong to the prior art. Specifically, RPN takes the output of the intermediate layer (i.e. the feature map) as input and outputs a set of region proposals, each with a score describing the probability that the region contains a person. A sliding window is used to sample potential regions, and for each sampling window, a classifier is used to check whether it closely matches a real person, and if so, RPN adjusts the boundaries of the window to make it more suitable. In this embodiment, a binary label can be assigned to each training window to indicate whether the region contains a person. To set the label, an intersection over union (IoU) indicator is defined, with the formula:

[0091]

[0092] where windows with IoU greater than 0.7 are set to positive (i.e. containing a person), and less than 0.3 are set to negative, and windows that do not meet the above two conditions are ignored in the training phase.

[0093] Step 4: Social intent recognition, complete the social grouping of the meeting scene.

[0094] Using the three-dimensional human posture data containing joint information output in step 3, estimate the distance between all objects in the region and the face direction, and construct a feature vector that can describe the correlation between two people. Assume two people r and t, the distance between them d and the rotation direction of the first person looking at the second person o rt , the rotation direction of the second person looking at the first person o tr to describe the relationship φ rt between the two, i.e. φ rt = (d, o rt , o tr ). The above distance d can be calculated using the spatial distance between the head center points of the two people, while o rt , o tr represent the angle of view, which can be calculated using the angle between the forward direction of the current person and the line connecting the head center points of the two people. The forward direction is the sagittal plane direction.

[0095] Combine all the people in the entire meeting space two by two, and the feature vector φ rtThe trained structural support vector machine is inputted for binary classification to obtain a classification label of whether two persons in the combination are in a social communication state. Based on the classification labels between two persons in the entire conference space domain, all persons are socially grouped according to the social communication relationship between each other, so as to complete the social intention recognition.

[0096] The structural support vector machine (SSVM) is a variant algorithm of the support vector machine (SVM) and is widely applied in multiple fields. The SSVM is superior to other SVM algorithms in classification performance and classification efficiency. The specific structure and principle of the SSVM belong to the prior art and can be implemented with reference to the prior art.

[0097] For easier understanding, the specific implementation is simply described in the embodiment. In order to describe the relationship between all personnel combinations in batches, for a given group of persons x, the pairwise relationship of them can be described by an affinity matrix W, if W rt > 0, two persons r and t are in the same group; if W rt < 0, they belong to different clusters. The relevant clustering y of a group of persons x is a partition that maximizes the affinity of item pairs in the same cluster, that is,

[0098]

[0099] where W rt is modeled as a linear combination of pairwise features of direction and distance over a time window, which can be constructed by the SSVM.

[0100] Let x i be a set of distance and direction features of a group of persons, y i be their clustering scheme, and w be a weight vector. Given input-output pair samples S = {(x1, y1), …, (x n , y n )}, the classification learning maps the input space X to the structured output space Y by a discriminant function F, and the prediction result is:

[0101]

[0102]

[0103] Solving the parameter problem of the structural support vector machine obtains:

[0104]

[0105]

[0106] where Φψ i (y) = ψ(x i , y i )- ψ(x i , y), ξ i is the slack variable, and Δ(y, y i ) is the loss function.

[0107] Using the cutting plane algorithm, iteratively find the most violated constraint Re-optimization until convergence. is represented as:

[0108]

[0109] The loss function uses the MITRE loss A tree can be used to represent a cluster. Consider two clustering solutions y, and their respective trees Q, P. The connected components of Q and P are denoted by Q i , i = 1, …, n and P i , i = 1, …, m. Only considering the relationships found in Q i , one can also find the relationships connecting the same in partition P, resulting in a set of subgroups defined as p(Q i ), define |Q i | as the number of people in Q i , then the global recall of Q is:

[0110]

[0111] The precision of Q can be calculated by swapping Q and P. Thus the loss is defined as

[0112]

[0113] Therefore, after training based on the above SSVM, the trained model can be used for inference to obtain the classification labels between two people in the entire conference space domain, which is used to realize social grouping.

[0114] In the fifth step, the obtained three-dimensional human posture recognition and social intention recognition results are visualized, the number of people and the three-dimensional posture of all objects in the detection area are displayed by using Unity3D, and different social groups are distinguished by different colors.

[0115] The above-described embodiments are only the preferred ones of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the protection scope of the present application.

Claims

1. A method for social intention recognition based on millimeter wave radar, characterized in that: include: S1. Acquire the millimeter-wave radar echo signal reflected by human motion modulation within the entire target area; S2, performing signal processing on the millimeter-wave radar echo signal to remove noise and interference, thereby obtaining a radio frequency tensor containing human posture information; S3, inputting the radio frequency tensor into a trained human posture recognition model to obtain a human target detection frame corresponding to each person within the target area and a three-dimensional human posture composed of human joint points; S4. Determine the head center position and forward viewing direction of each person according to the three-dimensional human posture of each person within the target area, then traverse the person combinations within the target area in pairs, calculate the spatial distance, first sight line angle, and second sight line angle between each pair of person combinations, and use the three calculated information as the feature vector of the pair of person combinations, where the spatial distance is the head center distance between the two people, the first sight line angle is the angle between the forward viewing direction of the first person and the line connecting the head centers of the two people, and the second sight line angle is the angle between the forward viewing direction of the second person and the line connecting the head centers of the two people; finally, input the feature vector of each pair of person combinations into the trained classification network for binary classification to obtain a classification label of whether the two people in the combination are in a social communication state; based on the classification labels between each two people within the entire target area, all people are socially grouped according to their social communication relationship, thereby completing social intention recognition.

2. The method for social intention recognition based on millimeter wave radar according to claim 1, characterized in that: The human posture recognition model includes a feature network, an RPN network and a posture estimation network. The radio frequency tensor is first input into the feature network to extract a feature map of the radio frequency signal, and then the RPN network extracts the region of interest from the feature map. Finally, the feature map extracted from the feature network and the region of interest extracted from the RPN network are input into the posture estimation network, and the regional feature map of each region of interest is obtained through the interest region pooling layer. After passing through the fully connected network, classification and regression are performed to obtain the position, category and accurate human target detection frame of the human joint points in each region of interest.

3. The method for social intention recognition based on millimeter wave radar according to claim 1, characterized in that: The human posture recognition model is pre-trained using supervised data, and the supervised data used for training is constructed as follows: The millimeter-wave radar echo signal reflected by human motion within the entire target area is obtained by the millimeter-wave radar, and the RGB image of the entire target area is simultaneously obtained by the visible light camera. The RGB image is pre-extracted with the OpenPose model to extract the 3D skeleton information including the position and category of the joint points and use it as the supervision label of the millimeter-wave radar echo signal.

4. The method for social intention recognition based on millimeter wave radar according to claim 1, characterized in that: The classification network adopts a structural support vector machine.

5. The method for social intention recognition based on millimeter wave radar according to claim 1, characterized in that: Also includes: S5. Visualize the human target detection frame and three-dimensional human posture corresponding to each person within the target area identified in S3 on the image, and use different tags to distinguish the information of people belonging to different social groups.

6. A social intention recognition system based on millimeter wave radar, characterized in that: include: Radar data acquisition module, used to obtain millimeter-wave radar echo signals reflected by human motion modulation within the entire target area; A signal processing module is used to process the millimeter-wave radar echo signal to remove noise and interference, thereby obtaining a radio frequency tensor containing human posture information; A posture recognition module is used to input the radio frequency tensor into a trained human posture recognition model to obtain a human target detection frame corresponding to each person in the target area and a three-dimensional human posture composed of human joint points; The social intention recognition module is used to determine the position of the head center point and the forward viewing direction of each person based on the three-dimensional human posture of each person within the target area, and then traverse the person combinations within the target area in pairs, calculate the spatial distance, first sight angle and second sight angle between each pair of person combinations, and use the three calculated information as the feature vector of the pair of person combinations. The spatial distance is the distance between the head centers of the two people, the first sight angle is the angle between the forward viewing direction of the first person and the line connecting the head centers of the two people, and the second sight angle is the angle between the forward viewing direction of the second person and the line connecting the head centers of the two people; finally, the feature vector of each pair of person combinations is input into the trained classification network for binary classification to obtain a classification label of whether the two people in the combination are in a social communication state; based on the classification labels between each pair of people within the entire target area, all people are socially grouped according to their social communication relationship, thereby completing social intention recognition.

7. The millimeter-wave radar-based social intention recognition system according to claim 6, characterized in that: The human posture recognition model includes a feature network, an RPN network and a posture estimation network. The radio frequency tensor is first input into the feature network to extract a feature map of the radio frequency signal, and then the RPN network extracts the region of interest from the feature map. Finally, the feature map extracted from the feature network and the region of interest extracted from the RPN network are input into the posture estimation network, and the regional feature map of each region of interest is obtained through the interest region pooling layer. After passing through the fully connected network, classification and regression are performed to obtain the position, category and accurate human target detection frame of the human joint points in each region of interest.

8. The millimeter-wave radar-based social intention recognition system according to claim 6, wherein: The human posture recognition model is pre-trained using supervised data, and the supervised data used for training is constructed as follows: The millimeter-wave radar echo signal reflected by human motion within the entire target area is obtained by the millimeter-wave radar, and the RGB image of the entire target area is simultaneously obtained by the visible light camera. The RGB image is pre-extracted with the OpenPose model to extract the 3D skeleton information including the position and category of the joint points and use it as the supervision label of the millimeter-wave radar echo signal.

9. The millimeter-wave radar-based social intention recognition system according to claim 6, wherein: The classification network adopts a structural support vector machine.

10. The millimeter-wave radar-based social intention recognition system according to claim 6, characterized in that: Also includes: The visualization module is used to visualize the human target detection frame and three-dimensional human posture corresponding to each person in the target area identified by the posture recognition module on the image, and use different tags to distinguish the information of people belonging to different social groups.

Images