Human body behavior recognition method and system based on millimeter wave integration and electronic equipment

Abstract

The invention discloses a human body behavior recognition method and system based on millimeter wave integration and electronic equipment. The method comprises the following steps: acquiring first millimeter wave signals of a plurality of radar sensors; performing transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal; feature merging is carried out on the first millimeter wave signal and the second millimeter wave signal through a preset neural network model, a merged feature is obtained, and the neural network model comprises a structure search module and an output module; inputting the combined features into a structure search module to enable the structure search module to perform continuous coding operation on the combined features, and outputting feature data; performing behavior prediction on the feature data through an output module to obtain an existence probability vector and a behavior probability vector corresponding to a prediction target; and determining the action type of the human body action according to the existence probability vector and the behavior probability vector. According to the embodiment of the invention, the accuracy of human body action recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, system and electronic device for human behavior recognition based on millimeter wave integration. Background Art

[0002] Multi-target human behavior recognition technology based on millimeter wave data can capture reflected waves through millimeter wave radar sensors and analyze human movements, such as judging whether a person falls, stands up, and other common movements. This technology uses the high penetration of millimeter waves to effectively monitor in environments with obstructed vision and has little interference with light and weather changes. It helps protect personal privacy by providing radar point cloud data instead of clear images, and is mainly used in security monitoring, elderly care, and smart home systems.

[0003] However, the point cloud data generated by low-cost millimeter-wave radar systems are often sparse and non-uniform, which poses a challenge to training fine-grained and accurate activity classifiers. Due to the sparsity of the data, all necessary action details may not be captured, thus affecting the accuracy of recognition. Summary of the invention

[0004] The embodiments of the present application provide a human behavior recognition method, system, electronic device and storage medium based on millimeter wave integration, which can improve the accuracy of human action recognition.

[0005] In a first aspect, an embodiment of the present application provides a method for human behavior recognition based on millimeter wave integration, the method comprising:

[0006] Acquire first millimeter wave signals of a plurality of radar sensors, where the first millimeter wave signals are obtained by capturing reflected waves of the radar sensors reflected by human body movements;

[0007] performing a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal;

[0008] Merging features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a merged feature, wherein the neural network model includes a structure search module and an output module;

[0009] Inputting the combined features into the structure search module, so that the structure search module performs continuous encoding operations on the combined features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements;

[0010] Performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target;

[0011] The action category of the human action is determined according to the existence probability vector and the behavior probability vector.

[0012] In some embodiments, the neural network model includes a first convolution layer and a second convolution layer; the step of merging features of the first millimeter wave signal and the second millimeter wave signal using a preset neural network model to obtain a merged feature includes:

[0013] Inputting the first millimeter wave signal and the second millimeter wave signal into a preset neural network model, so that the first convolution layer performs a feature encoding operation on the first millimeter wave signal and outputs first feature data, and the second convolution layer performs a feature encoding operation on the second millimeter wave signal and outputs second feature data;

[0014] The first feature data and the second feature data are combined to obtain a combined feature.

[0015] In some embodiments, the structure search module includes a fully connected layer and a third convolutional layer; the step of inputting the merged features into the structure search module so that the structure search module performs a continuous encoding operation on the merged features and outputs feature data includes:

[0016] Performing feature aggregation on the merged features through the structure search module to obtain a first intermediate feature;

[0017] Inputting the intermediate feature into the fully connected layer, so that the fully connected layer performs an encoding operation on the intermediate feature based on a preset search coefficient and outputs an encoded feature;

[0018] Performing a transposition operation on the encoded feature to obtain a second intermediate feature;

[0019] Inputting the second intermediate feature into the third convolution layer for feature capture, and outputting convolution feature information, wherein the convolution feature information includes the total number of targets of the predicted target;

[0020] A transposition operation is performed on the convolution feature information to output feature data.

[0021] In some embodiments, the method further comprises:

[0022] Regularizing the search coefficients;

[0023] The output connection of each neuron in the fully connected layer is weighted by the search coefficient after regularization constraint.

[0024] In some embodiments, the output module includes searching a fully connected layer; and performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target includes:

[0025] Performing a feature separation operation on the feature data through the output module to obtain a feature vector corresponding to the predicted target;

[0026] For each of the feature vectors, input the feature vector into the search fully connected layer for continuous encoding operation, and output a third intermediate feature;

[0027] The output module performs a full connection operation on the third intermediate feature to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

[0028] In some embodiments, determining the action category of the human action according to the existence probability vector and the behavior probability vector includes:

[0029] Performing a summing and averaging operation on all the existence probability vectors and the behavior probability vectors to obtain a probability vector;

[0030] Comparing all the probability vectors to select the largest target probability vector;

[0031] An action category corresponding to the human body action is determined according to the target probability vector.

[0032] In some embodiments, the method further comprises:

[0033] Performing mutual supervision learning on the structure search module through a preset mean square error loss function;

[0034] The output module is trained for human behavior using a preset cross entropy loss function, and the output module is subjected to mutual supervision learning using a preset divergence distance loss function.

[0035] In a second aspect, the embodiment of the present application further provides a human behavior recognition system based on millimeter wave integration, the system comprising:

[0036] A data acquisition module, used to acquire first millimeter wave signals of multiple radar sensors, where the first millimeter wave signals are captured by reflected waves reflected by the radar sensors from human body movements;

[0037] a signal transposition module, configured to perform a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal;

[0038] A feature merging module, used to merge the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a merged feature, wherein the neural network model includes a structure search module and an output module;

[0039] A continuous encoding module, used for inputting the combined features into the structure search module, so that the structure search module performs a continuous encoding operation on the combined features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements;

[0040] A behavior prediction module, used to perform behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target;

[0041] The category determination module is used to determine the action category of the human action according to the existence probability vector and the behavior probability vector.

[0042] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for human behavior recognition based on millimeter wave integration as described in the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the human behavior recognition method based on millimeter wave integration as described in the first aspect.

[0044] The human behavior recognition method based on millimeter wave integration provided in the embodiment of the present application has at least the following beneficial effects: first, the first millimeter wave signals of multiple radar sensors are acquired, so that millimeter wave signals at different angles can be obtained, which is convenient for the subsequent analysis of human body movements through the captured reflected waves, and then the first millimeter wave signal is transposed to change the input dimension of the millimeter wave signal to obtain the second millimeter wave signal. The first millimeter wave signal can be converted into another input dimension to obtain two millimeter wave signals of different dimensions. The first millimeter wave signal and the second millimeter wave signal are feature merged through a preset neural network model, that is, the features at different dimensional levels are encoded into feature data to obtain merged features. By performing feature merging, the most relevant features can be extracted and selected, the feature dimensions can be reduced, the complexity of the model can be reduced, the risk of overfitting can be avoided, and the training efficiency and prediction performance of the model can be improved. After that, the merged features are input into the structure search module so that the structure search module can continuously encode the merged features, realize the mining of potential correlations between feature values, and output feature data, so as to capture more action details, facilitate the subsequent human behavior recognition, improve the accuracy of human behavior recognition, and obtain more discriminative and expressive features by constructing and combining new features. Then, the output module performs behavior prediction on the feature data to obtain the existence probability vector and behavior probability vector corresponding to the prediction target, so as to determine whether the behavior corresponding to the prediction target exists and its behavior category, and improve the accuracy of specific behavior recognition. Finally, the action category of the human action is determined according to the existence probability vector and the behavior probability vector, and the accuracy of human action recognition is improved.

[0045] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the examples of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0047] Figure 1 It is a flowchart of a specific method of the human behavior recognition method based on millimeter wave integration provided in an embodiment of the present application;

[0048] Figure 2 It is a specific flow chart of merging the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model provided in an embodiment of the present application;

[0049] Figure 3It is a specific flow chart of the structure search module provided in the embodiment of the present application performing continuous encoding operation on the merged features;

[0050] Figure 4 is a specific flow chart of a human behavior recognition method based on millimeter wave integration provided in another embodiment of the present application;

[0051] Figure 5 This is a specific flow chart of predicting behavior of feature data through an output module provided in an embodiment of the present application;

[0052] Figure 6 It is a specific flow chart of determining the action category of a human action according to an existence probability vector and a behavior probability vector provided by an embodiment of the present application;

[0053] Figure 7 is a specific flow chart of a human behavior recognition method based on millimeter wave integration provided in another embodiment of the present application;

[0054] Figure 8 is a schematic diagram of the structure of a human behavior recognition system based on millimeter wave integration provided in an embodiment of the present application;

[0055] Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application;

[0056] Fig.10 is a module schematic diagram of an input module provided in an example of the present application;

[0057] Fig.11 is a module schematic diagram of a structure search module provided in an example of the present application;

[0058] Fig.12 is a module schematic diagram of a structure search module provided in an example of the present application;

[0059] Fig.13 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] The human behavior recognition method based on millimeter wave integration provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms; the software can be an application that implements the above method, etc., but is not limited to the above forms.

[0064] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0065] Multi-target human behavior recognition technology based on millimeter wave data can capture reflected waves through millimeter wave radar sensors and analyze human movements, such as judging whether a person falls, stands up, and other common movements. This technology uses the high penetration of millimeter waves to effectively monitor in environments with obstructed vision and has little interference with light and weather changes. It helps protect personal privacy by providing radar point cloud data instead of clear images, and is mainly used in security monitoring, elderly care, and smart home systems.

[0066] However, the point cloud data generated by low-cost millimeter-wave radar systems are often sparse and non-uniform, which poses a challenge to training fine-grained and accurate activity classifiers. Due to the sparsity of the data, all necessary action details may not be captured, thus affecting the accuracy of recognition.

[0067] In order to solve the above problems, the embodiments of the present application provide a human behavior recognition method, system, electronic device and storage medium based on millimeter wave integration. First, the first millimeter wave signal of multiple radar sensors is obtained, so that millimeter wave signals at different angles can be obtained, which is convenient for subsequent analysis of human body movements through the captured reflected waves. Then, the first millimeter wave signal is transposed to change the input dimension of the millimeter wave signal to obtain a second millimeter wave signal. The first millimeter wave signal can be converted into another input dimension to obtain two millimeter wave signals of different dimensions. The first millimeter wave signal and the second millimeter wave signal are feature merged through a preset neural network model, that is, features at different dimensional levels are encoded into feature data to obtain merged features. By performing feature merging, the most relevant features can be extracted and selected, the feature dimensions can be reduced, the complexity of the model can be reduced, the risk of overfitting can be avoided, and the training efficiency and prediction performance of the model can be improved. After that, the merged features are input into the structure search module so that the structure search module can continuously encode the merged features, realize the mining of potential correlations between feature values, and output feature data, so as to capture more action details, facilitate the subsequent human behavior recognition, improve the accuracy of human behavior recognition, and obtain more discriminative and expressive features by constructing and combining new features. Then, the output module performs behavior prediction on the feature data to obtain the existence probability vector and behavior probability vector corresponding to the prediction target, so as to determine whether the behavior corresponding to the prediction target exists and its behavior category, and improve the accuracy of specific behavior recognition. Finally, the action category of the human action is determined according to the existence probability vector and the behavior probability vector, and the accuracy of human action recognition is improved.

[0068] Reference Figure 1 , Figure 14 is a flowchart of a specific method of a human behavior recognition method based on millimeter wave integration provided in an embodiment of the present application. In some embodiments, the human behavior recognition method based on millimeter wave integration includes but is not limited to steps S101 to S106.

[0069] Step S101, obtaining first millimeter wave signals of multiple radar sensors, where the first millimeter wave signals are captured by reflection waves reflected by the radar sensors from human body movements.

[0070] In step S101 of some embodiments, the radar sensor captures the reflected waves reflected by human body movements. The embodiment of the present application first obtains the first millimeter wave signals of multiple radar sensors, wherein the first millimeter wave signals in the embodiment of the present application are millimeter wave signals at different angles, thereby achieving comprehensive capture of millimeter wave signals in different directions, and facilitating subsequent analysis of human body movements through the first millimeter wave signal, such as determining common movements such as falling or standing up.

[0071] It is understandable that the high penetration of millimeter waves can effectively monitor in environments with obstructed vision, and is less affected by changes in lighting and weather. And it helps protect personal privacy by providing radar point cloud data rather than clear images.

[0072] Step S102: performing a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal.

[0073] In step S102 of some embodiments, after acquiring the first millimeter wave signal, a transposition operation is performed on the first millimeter wave signal to convert the first millimeter wave signal into another input dimension to obtain a second millimeter wave signal. Specifically, the embodiment of the present application converts the millimeter wave signal into another input dimension through a matrix transposition operation. The matrix transposition operation facilitates the subsequent model to learn the characteristics of the data from different angles and dimensions, which helps to improve the model's generalization ability for different actions, so that it can maintain a high recognition accuracy when facing new and unseen actions.

[0074] Step S103, combining features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a combined feature, wherein the neural network model includes a structure search module and an output module.

[0075] In step S103 of some embodiments, features of the first millimeter wave signal and the second millimeter wave signal are merged through a preset neural network model. By constructing and combining new features, more discriminative and expressive merged features can be obtained. These new features can often reveal hidden patterns and correlations in the data, thereby improving the predictive performance of the model and avoiding the inability to capture all necessary action details due to data sparsity.

[0076] Step S104: input the merged features into a structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements.

[0077] In step S104 of some embodiments, the merged features are input into the structure search module so that the structure search module performs continuous encoding operations on the merged features to achieve efficient encoding of the features, thereby being able to mine the potential correlation between feature values ​​and output feature vectors of all predicted targets to obtain feature data to facilitate subsequent behavior prediction.

[0078] Step S105, performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target.

[0079] In step S105 of some embodiments, the output module performs behavior prediction on the feature data to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target, wherein the existence probability vector and the behavior probability vector can provide the model with quantitative information about whether an action exists and the specific behavior category. Specifically, the existence probability vector can help identify whether a specific action exists, while the behavior probability vector can be used to classify the specific category of the action. The existence probability vector and the behavior probability vector can improve the recognition accuracy of specific action categories, or reduce the misrecognition of certain confusing actions.

[0080] Step S106, determining the action category of the human action according to the existence probability vector and the behavior probability vector.

[0081] In step S106 of some embodiments, it is determined whether the target human body exists and to which behavior category the target human body belongs based on the existence probability vector and the behavior probability vector, and the action category of the human body action is determined to achieve accurate recognition of the human body action, thereby improving the accuracy of human body action recognition.

[0082] Reference Figure 2 , Figure 2 It is a specific flow chart of merging the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model provided in an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S201 to S202.

[0083] It should be noted that the neural network model includes an input module, each input module includes a first convolutional layer and a second convolutional layer, wherein the first convolutional layer is a 1*3 convolutional layer, including a 1*3 group of convolution operations, a batch normalization operation, and a Relu activation operation, and the second convolutional layer is a 1*1 convolutional layer, including a 1*1 group of convolution operations, a batch normalization operation, and a Relu activation operation.

[0084] Step S201, input the first millimeter wave signal and the second millimeter wave signal into a preset neural network model, so that the first convolution layer performs a feature encoding operation on the first millimeter wave signal and outputs first feature data, and the second convolution layer performs a feature encoding operation on the second millimeter wave signal and outputs second feature data.

[0085] Step S202: Merge the first feature data and the second feature data to obtain a merged feature.

[0086] In steps S201 to S202 of some embodiments, in the process of merging features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model, first, the first millimeter wave signal and the second millimeter wave signal are input into the preset neural network model so that the first convolution layer performs feature encoding operation on the first millimeter wave signal and outputs first feature data, and the second convolution layer performs feature encoding operation on the second millimeter wave signal and outputs second feature data. The first convolution layer and the second convolution layer mainly perform feature encoding of the input data at different dimensional levels at the minimum computational cost and generate feature data. Thereafter, the first feature data and the second feature data are feature merged to obtain merged features. By constructing and combining new features, more discriminative and expressive features can be obtained. These new features can often reveal hidden patterns and correlations in the data, thereby improving the prediction performance of the model and avoiding the inability to capture all necessary action details due to data sparsity.

[0087] In some embodiments, the embodiments of the present application can establish communication with other input modules by minimizing the mean squared error (MSE) loss function, and perform mutual supervised learning on each other's output features.

[0088] Specifically, the minimum mean square error loss function is expressed as follows:

[0089]

[0090] Among them, in the loss function Medium, F i and F j denote the output feature maps of the i-th and j-th input modules respectively, and k denotes the total number of input modules.

[0091] Reference Figure 3 , Figure 3 It is a specific flow chart of the structure search module provided in the embodiment of the present application performing continuous encoding operation on the merged features. In some embodiments, the method includes but is not limited to steps S301 to S305.

[0092] It should be noted that the structure search module includes a fully connected layer and a third convolutional layer, wherein the structure search module in the embodiment of the present application includes five fully connected layers.

[0093] Step S301: perform feature aggregation on the merged features through a structure search module to obtain a first intermediate feature.

[0094] Step S302: input the intermediate features into the fully connected layer, so that the fully connected layer performs encoding operations on the intermediate features based on preset search coefficients and outputs encoded features.

[0095] Step S303, performing a transposition operation on the encoded feature to obtain a second intermediate feature.

[0096] Step S304: input the second intermediate feature into the third convolution layer for feature capture, and output convolution feature information, wherein the convolution feature information includes the total number of targets of the predicted target.

[0097] Step S305, performing a transposition operation on the convolution feature information and outputting feature data.

[0098] In steps S301 to S305 of some embodiments, in the process of performing continuous encoding operations on the merged features through the structure search module, first, the merged features are aggregated through the structure search module to obtain a first intermediate feature. Specifically, the row features are averaged and summed through the aggregation operation to obtain the first intermediate feature, and then the intermediate feature is input into the fully connected layer, so that the fully connected layer performs an encoding operation on the intermediate feature based on a preset search coefficient. Specifically, the fully connected layer performs a structured full connection operation, a batch normalization operation, and a Relu activation operation on the intermediate feature, and outputs the encoded feature. In addition, the structured full connection operation in the embodiment of the present application adds a search coefficient compared to the ordinary full connection operation, which is used for All output connections of each neuron are weighted to measure the importance of a neuron, so that the potential correlation between feature values ​​can be mined and efficient encoding of features can be achieved. After that, the encoded features are transposed to obtain the second intermediate features to achieve dimensional conversion of the encoded features. The second intermediate features are then input into the third convolutional layer for feature capture and convolution feature information is output, wherein the convolution feature information includes the total number of predicted targets, which is convenient for subsequent behavior recognition based on the convolution feature information. Finally, the convolution feature information is transposed to output feature data, so that the potential correlation between feature values ​​can be mined and the feature vectors of all predicted targets are output to obtain feature data for subsequent behavior prediction.

[0099] Reference Figure 4 , Figure 4It is a specific flow chart of a human behavior recognition method based on millimeter wave integration provided by another embodiment of the present application. In some embodiments, the method includes but is not limited to step S401 to step S402.

[0100] Step S401, regularization constraints are applied to the search coefficients.

[0101] Step S402, weighting the output connections of the neurons in each fully connected layer by the search coefficient after regularization constraints.

[0102] In steps S401 to S402 of some embodiments, the embodiments of the present application also perform regularization constraints on the search coefficients to make their values ​​sparse, so as to capture more feature details, and then weight the output connections of the neurons in each fully connected layer with the regularized search coefficients to measure the importance of different neurons.

[0103] In some embodiments, after the output connections of the neurons in each fully connected layer are weighted by the search coefficient after regularization constraints, for neurons with smaller search coefficients, it means that the output connections of the neurons contribute very little to the entire network and can be directly pruned. Through this mechanism, the embodiments of the present application can obtain a more compact and concise network structure and effectively reduce the amount of fully connected calculations.

[0104] Reference Figure 5 , Figure 5 It is a specific flow chart of predicting behavior of feature data through an output module provided in an embodiment of the present application. In some embodiments, the method includes but is not limited to steps S501 to S503.

[0105] It should be noted that the output module includes a search fully connected layer, wherein the output module in the embodiment of the present application includes two search fully connected layers.

[0106] Step S501, performing feature separation operation on feature data through an output module to obtain a feature vector corresponding to a prediction target.

[0107] Step S502: for each feature vector, input the feature vector into the search fully connected layer for continuous encoding operation, and output the third intermediate feature.

[0108] Step S503: Perform a full connection operation on the third intermediate feature through the output module to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

[0109] In steps S501 to S503 of some embodiments, the output module is mainly responsible for receiving feature data from the search module, and globally encoding the feature vector of each prediction target and generating the final prediction result. In the process of predicting the behavior of the feature data through the output module, the feature data is firstly subjected to a feature separation operation through the output module to obtain a feature vector of a single prediction target, that is, a feature vector corresponding to the prediction target is obtained. For each feature vector, the feature vector is input into the search fully connected layer for continuous encoding operation. Specifically, the feature vector is continuously encoded through two search fully connected layers, and intermediate features of the same dimension are obtained, and the third intermediate feature is output. Finally, the third intermediate feature is fully connected through the output module to generate an existence probability vector and a behavior probability vector, wherein the existence probability vector and the behavior probability vector can provide the model with quantitative information about whether an action exists and a specific behavior category. Specifically, the existence probability vector can help identify whether a specific action exists, and the behavior probability vector can be used to classify the specific category of the action. The existence probability vector and the behavior probability vector can improve the recognition accuracy of a specific action category, or reduce the misrecognition of some easily confused actions.

[0110] Reference Figure 6 , Figure 6 It is a specific flow chart of determining the action category of a human action according to the existence probability vector and the behavior probability vector provided by the embodiment of the present application. In some embodiments, the method includes but is not limited to steps S601 to S603.

[0111] Step S601, performing a summing and averaging operation on all existence probability vectors and behavior probability vectors to obtain a probability vector.

[0112] Step S602: compare all probability vectors to select the largest target probability vector.

[0113] Step S603: determining the action category corresponding to the human body action according to the target probability vector.

[0114] In steps S601 to S603 of some embodiments, in the process of determining the action category of human body actions based on the existence probability vector and the behavior probability vector, the embodiments of the present application first perform a sum and average operation on all existence probability vectors and behavior probability vectors to obtain multiple probability vectors, and then compare all probability vectors to screen out the largest target probability vector. Thereafter, it is determined whether the target human body exists and to which behavior category it belongs based on the existence probability vector and the behavior probability vector corresponding to the target probability vector, thereby achieving accurate recognition of human body actions and improving the accuracy of human body action recognition.

[0115] Reference Figure 7 , Figure 7701 to 702.

[0116] Step S701, performing mutual supervision learning on the structure search module through a preset mean square error loss function.

[0117] In step S701 of some embodiments, the embodiments of the present application will also perform mutual supervised learning on the structure search modules through a preset mean square error loss function. Specifically, during the training process, each structure search module performs mutual supervised learning with other search modules by minimizing the mean square error loss function, and can perform mutual supervised learning through intermediate features or probability vectors of different modules to jointly improve the optimal solution of the network.

[0118] It can be understood that the mean square error loss function is expressed as follows:

[0119]

[0120] In the loss function Medium, F u and F v They represent the output feature maps of the u-th and v-th structure search modules respectively, and table l represents the total number of structure search modules.

[0121] Step S702, human behavior training is performed on the output module through a preset cross entropy loss function, and mutual supervision learning is performed on the output module through a preset divergence distance loss function.

[0122] In step S702 of some embodiments, the embodiments of the present application will also perform human behavior training on the output module through a preset cross entropy loss function to improve the behavior prediction accuracy of the neural network model, and perform mutual supervised learning on the output modules through a preset divergence distance loss function. By performing mutual supervised learning on the output modules, the behavior probability distribution of other output modules is learned to improve the optimal solution of the network.

[0123] Specifically, the cross-entropy loss function of the embodiment of the present application includes a first cross-entropy loss function and a second cross-entropy loss function, wherein the first cross-entropy loss function is used to learn the target existence task, so that the existence probability vector generated by the output module is more accurate, and the second cross-entropy loss function is used to learn the behavior recognition task, so that the behavior probability vector of the output module is more accurate.

[0124] It can be understood that the first cross entropy loss function is expressed as follows:

[0125]

[0126] Among them, yi and Represent the true label and predicted probability of whether the target exists or not, respectively.

[0127] The second cross entropy loss function is expressed as follows:

[0128]

[0129] Among them, b i and They represent the true label and predicted probability of the target’s current behavior respectively, and B represents the total number of behavior categories. Common human behavior categories include falling, standing, lying still, etc.

[0130] The divergence distance loss function is expressed as follows:

[0131]

[0132] in, and denote the predicted probabilities of the i-th behavior of the k-th and h-th output modules respectively, and K denotes the total number of output modules.

[0133] See also Figure 8 The embodiment of the present application also provides a human behavior recognition system based on millimeter wave integration, which can implement the above-mentioned human behavior recognition method based on millimeter wave integration, and the system includes:

[0134] A data acquisition module 801 is used to acquire first millimeter wave signals of multiple radar sensors, where the first millimeter wave signals are captured by reflected waves reflected by the radar sensors from human body movements;

[0135] A signal transposition module 802, configured to perform a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal;

[0136] A feature merging module 803 is used to merge the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a merged feature, wherein the neural network model includes a structure search module and an output module;

[0137] A continuous encoding module 804 is used to input the merged features into the structure search module, so that the structure search module performs a continuous encoding operation on the merged features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements;

[0138] The behavior prediction module 805 is used to perform behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target;

[0139] The category determination module 806 is used to determine the action category of the human action according to the existence probability vector and the behavior probability vector.

[0140] In order to more clearly explain the human behavior recognition method, system and electronic device based on millimeter wave integration according to the embodiments of the present application, specific examples are given below for illustration.

[0141] Example 1:

[0142] See also Fig. 9 , Fig. 9 It is a schematic diagram of the overall framework provided by an example of this application.

[0143] like Fig. 9 As shown, the embodiment of the present application can receive millimeter wave signals at different angles. It is assumed here that the dimension of each millimeter wave signal is: 128*5, where 128 represents the number of radar point clouds, and 5 represents the number of features, which respectively represent the spatial coordinates (x, y, z), signal transmission rate and reflection intensity of each point. The millimeter wave signal at each angle passes through the input module, the structure search module, and the output module in turn, and finally generates the existence probability vector (1*2) and behavior probability vector (1*B) of N targets, where B represents the total number of behavior categories. During the training process, the intermediate features or probability vectors of different modules supervise each other and jointly improve the optimal solution of the network. Finally, a final prediction result is obtained by voting on all output results.

[0144] The millimeter wave integrated human behavior recognition method of this example is specifically described below in combination with different modules.

[0145] See also Fig.10 , Fig.10 This is a module diagram of an input module provided in an example of the present application.

[0146] In some embodiments, the input module adopts a dual input structure and transforms the millimeter wave signal into another input dimension: 5*128 through a matrix transposition operation. The two different forms of millimeter wave data will pass through a 1*3 convolution layer (including 1*3 groups of convolution operations, batch normalization operations, and Relu activation operations) and a 1*1 convolution layer (including 1*1 convolution operations, batch normalization operations, and Relu activation operations) respectively. They mainly encode the features of the input data at different dimensional levels with the minimum computational cost and generate 128*256 and 5*256 feature data. Finally, a merging operation is performed to obtain a 133*256 output feature. During the training process, each input module establishes communication with other input modules by minimizing the mean square error loss function, and performs mutual supervision learning on each other's output features.

[0147] See also Fig.11 , Fig.11It is a module diagram of a structure search module provided in an example of the present application.

[0148] In some embodiments, the structure search module, as the main structure of the model, is mainly responsible for continuously encoding the feature data of the input module, mining the potential correlation between feature values, and outputting the feature vectors of all predicted targets. In the structure search module, we focus on introducing a fully connected layer based on a structured search mechanism. The fully connected layer mainly includes a structured fully connected operation, a batch normalization operation, and a Relu activation operation. Compared with the ordinary fully connected operation, the structured fully connected operation adds a search coefficient to weight all output connections of each neuron to measure the importance of a neuron.

[0149] In network training, the embodiment of the present application performs L1 regularization constraints on all search coefficients to make their values ​​have sparse characteristics. Fig.11 As shown, W 3 =[0.8,0.4,0.001,0.09] represents the search coefficients of all neurons in the third fully connected layer, where the search coefficient of the third neuron is close to zero, indicating that its output connection contributes little to the entire network and can be directly pruned. Through this mechanism, we can obtain a more compact and concise network structure and effectively reduce the amount of full connection calculations. In the structural search module, the input features are first averaged and summed through the aggregation operation to obtain 1*256 intermediate features, and then the features are efficiently encoded through 5 fully connected layers based on the structured search mechanism, and finally the N*256 output features are generated through the transposition operation and the 1*1 convolution layer, where N represents the total number of predicted targets.

[0150] During the training process, each structure search module performs mutual supervision learning with other search modules by minimizing the mean square error loss function.

[0151] See also Fig.12 , Fig.12 It is a module diagram of a structure search module provided in an example of the present application.

[0152] In some embodiments, the output module is mainly responsible for receiving feature data from the search module, and globally encoding the feature vector of each prediction target and generating the final prediction result. In this module, the target feature map (N*256) first obtains the feature vector (1*256) of a single prediction target through a separation operation, and then continuously encodes through two fully connected layers based on the structured search mechanism and obtains intermediate features of the same dimension, and finally generates the existence probability vector (1*2) and the behavior probability vector (1*B) respectively through two fully connected operations.

[0153] During the training process, the model uses the cross entropy loss function to learn the target existence task. In the behavior recognition task learning, the model also uses the cross entropy loss function. In this module, in addition to learning the prediction target existence task and the behavior recognition task, the embodiment of the present application also introduces a loss function based on the KL (Kullback-Leibler Divergence, relative entropy) divergence distance to learn the behavior probability distribution of other output modules.

[0154] The method of the embodiment of the present application can not only utilize multi-angle millimeter wave data to effectively improve behavior recognition performance, but is also designed as a lightweight neural network model, which is more friendly to edge device deployment.

[0155] The millimeter wave integration-based human behavior recognition system of the embodiment of the present application is used to execute the millimeter wave integration-based human behavior recognition method in the above embodiment. Its specific processing process is the same as the millimeter wave integration-based human behavior recognition method in the above embodiment, and will not be repeated here.

[0156] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the human behavior recognition method based on millimeter wave integration in the above embodiment of the present application.

[0157] See also Fig.13 , Fig.13 The hardware structure of the electronic device provided in the embodiment of the present application is illustrated, and the electronic device includes:

[0158] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0159] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the human behavior recognition method based on millimeter wave integration in the embodiment of the present application;

[0160] Input / output interface 903, used to implement information input and output;

[0161] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0162] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0163] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0164] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the human behavior recognition method based on millimeter wave integration is implemented.

[0165] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0167] It can be understood by those skilled in the art that Figure 1-13 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0168] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0169] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0170] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0171] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0173] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0176] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A human behavior recognition method based on millimeter wave integration, characterized in that: The method comprises: Acquire first millimeter wave signals of a plurality of radar sensors, where the first millimeter wave signals are obtained by capturing reflected waves of the radar sensors reflected by human body movements; performing a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal; Merging features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a merged feature, wherein the neural network model includes a structure search module and an output module; Inputting the combined features into the structure search module, so that the structure search module performs continuous encoding operations on the combined features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements; Performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target; The action category of the human action is determined according to the existence probability vector and the behavior probability vector.

2. The method for human behavior recognition based on millimeter wave integration according to claim 1, characterized in that: The neural network model includes a first convolution layer and a second convolution layer; the feature merging of the first millimeter wave signal and the second millimeter wave signal by using a preset neural network model to obtain a merged feature includes: Inputting the first millimeter wave signal and the second millimeter wave signal into a preset neural network model, so that the first convolution layer performs a feature encoding operation on the first millimeter wave signal and outputs first feature data, and the second convolution layer performs a feature encoding operation on the second millimeter wave signal and outputs second feature data; The first feature data and the second feature data are combined to obtain a combined feature.

3. The method for human behavior recognition based on millimeter wave integration according to claim 1, characterized in that: The structure search module includes a fully connected layer and a third convolutional layer; the step of inputting the merged features into the structure search module so that the structure search module performs a continuous encoding operation on the merged features and outputs feature data includes: Performing feature aggregation on the merged features through the structure search module to obtain a first intermediate feature; Inputting the intermediate feature into the fully connected layer, so that the fully connected layer performs an encoding operation on the intermediate feature based on a preset search coefficient and outputs an encoded feature; Performing a transposition operation on the encoded feature to obtain a second intermediate feature; Inputting the second intermediate feature into the third convolution layer for feature capture, and outputting convolution feature information, wherein the convolution feature information includes the total number of targets of the predicted target; A transposition operation is performed on the convolution feature information to output feature data.

4. The method for human behavior recognition based on millimeter wave integration according to claim 3 is characterized in that: The method further comprises: Regularizing the search coefficients; The output connection of each neuron in the fully connected layer is weighted by the search coefficient after regularization constraint.

5. The method for human behavior recognition based on millimeter wave integration according to claim 1, characterized in that: The output module includes searching a fully connected layer; the output module performs behavior prediction on the feature data to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target, including: Performing a feature separation operation on the feature data through the output module to obtain a feature vector corresponding to the predicted target; For each of the feature vectors, input the feature vector into the search fully connected layer for continuous encoding operation, and output a third intermediate feature; The output module performs a full connection operation on the third intermediate feature to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

6. The method for human behavior recognition based on millimeter wave integration according to claim 1, characterized in that: The determining the action category of the human action according to the existence probability vector and the behavior probability vector comprises: Performing a summing and averaging operation on all the existence probability vectors and the behavior probability vectors to obtain a probability vector; Comparing all the probability vectors to select the largest target probability vector; An action category corresponding to the human body action is determined according to the target probability vector.

7. The method for human behavior recognition based on millimeter wave integration according to claim 1, characterized in that: The method further comprises: Performing mutual supervision learning on the structure search module through a preset mean square error loss function; The output module is trained for human behavior using a preset cross entropy loss function, and the output module is subjected to mutual supervision learning using a preset divergence distance loss function.

8. A human behavior recognition system based on millimeter wave integration, characterized in that: The system comprises: A data acquisition module, used to acquire first millimeter wave signals of multiple radar sensors, where the first millimeter wave signals are captured by reflected waves reflected by the radar sensors from human body movements; a signal transposition module, configured to perform a transposition operation on the first millimeter wave signal to obtain a second millimeter wave signal; A feature merging module, used to merge the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain a merged feature, wherein the neural network model includes a structure search module and an output module; A continuous encoding module, used for inputting the combined features into the structure search module, so that the structure search module performs a continuous encoding operation on the combined features and outputs feature data, wherein the feature data includes a plurality of predicted targets corresponding to human body movements; A behavior prediction module, used to perform behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target; The category determination module is used to determine the action category of the human action according to the existence probability vector and the behavior probability vector.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the human behavior recognition method based on millimeter wave integration as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the human behavior recognition method based on millimeter wave integration as described in any one of claims 1 to 7.

Images