Human behavior recognition method and system based on millimeter wave integration, and electronic device

Abstract

The application discloses a human body behavior recognition method and system based on millimeter wave integration and electronic equipment. The method comprises the following steps: acquiring first millimeter wave signals of multiple radar sensors; performing a transposition operation on the first millimeter wave signals to obtain second millimeter wave signals; performing feature merging on the first millimeter wave signals and the second millimeter wave signals through a preset neural network model to obtain merged features, wherein the neural network model comprises a structure search module and an output module; inputting the merged features into the structure search module to enable the structure search module to perform a continuous encoding operation on the merged features and output feature data; performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to a prediction target; and determining a motion category of a human body motion according to the existence probability vector and the behavior probability vector. In the embodiment of the application, the accuracy of human body motion recognition can be improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a human behavior recognition method, system and electronic device based on millimeter wave integration. Background Technology

[0002] Multi-target human behavior recognition technology based on millimeter-wave data can capture reflected waves using millimeter-wave radar sensors to analyze human movements, such as determining whether a person has fallen or is standing. This technology leverages the high penetration of millimeter waves, enabling effective monitoring even in environments with obstructed visibility and minimal interference from changes in lighting and weather. By providing radar point cloud data rather than clear images, it helps protect personal privacy and is primarily used in security monitoring, elderly care, and smart home systems.

[0003] However, the point cloud data generated by low-cost millimeter-wave radar systems is often sparse and uneven, which poses a challenge to training fine-grained and accurate activity classifiers. Due to the sparsity of the data, it may be impossible to capture all the necessary motion details, thus affecting the accuracy of recognition. Summary of the Invention

[0004] This application provides a method, system, electronic device, and storage medium for human behavior recognition based on millimeter wave integration, which can improve the accuracy of human action recognition.

[0005] In a first aspect, embodiments of this application provide a human behavior recognition method based on millimeter-wave integration, the method comprising:

[0006] The first millimeter-wave signal from multiple radar sensors is acquired, and the first millimeter-wave signal is obtained by capturing the reflected wave reflected by the radar sensor in response to human movement;

[0007] The first millimeter-wave signal is transposed to obtain the second millimeter-wave signal;

[0008] The first millimeter-wave signal and the second millimeter-wave signal are combined using a preset neural network model to obtain combined features. The neural network model includes a structure search module and an output module.

[0009] The merged features are input into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions;

[0010] The output module performs behavior prediction on the feature data to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target.

[0011] The action category of the human action is determined based on the existence probability vector and the behavior probability vector.

[0012] In some embodiments, the neural network model includes a first convolutional layer and a second convolutional layer; the step of combining the features of the first millimeter-wave signal and the second millimeter-wave signal using a preset neural network model to obtain combined features includes:

[0013] The first millimeter-wave signal and the second millimeter-wave signal are input into a preset neural network model, so that the first convolutional layer performs feature encoding operation on the first millimeter-wave signal and outputs first feature data, and the second convolutional layer performs feature encoding operation on the second millimeter-wave signal and outputs second feature data.

[0014] The first feature data and the second feature data are merged to obtain merged features.

[0015] In some embodiments, the structure search module includes a fully connected layer and a third convolutional layer; the step of inputting the merged features into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, includes:

[0016] The structure search module performs feature aggregation on the merged features to obtain the first intermediate feature;

[0017] The intermediate features are input into the fully connected layer, so that the fully connected layer encodes the intermediate features based on preset search coefficients and outputs encoded features;

[0018] The encoded feature is transposed to obtain the second intermediate feature;

[0019] The second intermediate feature is input into the third convolutional layer for feature capture, and convolutional feature information is output, wherein the convolutional feature information includes the total number of targets in the predicted target;

[0020] The convolutional feature information is transposed to output feature data.

[0021] In some embodiments, the method further includes:

[0022] Apply regularization constraints to the search coefficients;

[0023] The output connections of each neuron in the fully connected layer are weighted by search coefficients after regularization constraints.

[0024] In some embodiments, the output module includes a search fully connected layer; the step of performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target includes:

[0025] The output module performs feature separation on the feature data to obtain the feature vector corresponding to the prediction target;

[0026] For each feature vector, the feature vector is input into the search fully connected layer for continuous encoding operation, and the third intermediate feature is output.

[0027] The output module performs a fully connected operation on the third intermediate feature to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

[0028] In some embodiments, determining the action category of the human action based on the existence probability vector and the behavior probability vector includes:

[0029] The probability vector is obtained by summing and averaging all the existence probability vectors and the behavior probability vectors.

[0030] Compare all the probability vectors to select the target probability vector with the largest value;

[0031] The action category of the human body action is determined based on the target probability vector.

[0032] In some embodiments, the method further includes:

[0033] The structure search module is subjected to mutual supervision learning through a preset mean square error loss function;

[0034] The output module is trained on human behavior using a preset cross-entropy loss function, and the output module is subjected to mutual supervision learning using a preset divergence distance loss function.

[0035] Secondly, embodiments of this application also provide a human behavior recognition system based on millimeter-wave integration, the system comprising:

[0036] The data acquisition module is used to acquire the first millimeter-wave signals from multiple radar sensors, which are obtained by capturing the reflected waves from human body movements by the radar sensors.

[0037] The signal transpose module is used to transpose the first millimeter-wave signal to obtain the second millimeter-wave signal;

[0038] The feature merging module is used to merge the features of the first millimeter-wave signal and the second millimeter-wave signal through a preset neural network model to obtain merged features. The neural network model includes a structure search module and an output module.

[0039] A continuous encoding module is used to input the merged features into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions;

[0040] The behavior prediction module is used to perform behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target.

[0041] The category determination module is used to determine the action category of the human action based on the existence probability vector and the behavior probability vector.

[0042] Thirdly, embodiments of this application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the human behavior recognition method based on millimeter-wave integration as described in the first aspect.

[0043] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the millimeter-wave integrated human behavior recognition method as described in the first aspect.

[0044] The human behavior recognition method based on millimeter wave integration provided in this application has at least the following beneficial effects: First, it acquires first millimeter wave signals from multiple radar sensors, thereby obtaining millimeter wave signals from different angles, which facilitates subsequent analysis of human actions through captured reflected waves. Then, it transposes the first millimeter wave signal to change the input dimension of the millimeter wave signal, obtaining a second millimeter wave signal. This transforms the first millimeter wave signal into another input dimension, resulting in two millimeter wave signals of different dimensions. The first and second millimeter wave signals are then feature-merged using a preset neural network model, that is, features at different dimensional levels are encoded into feature data to obtain merged features. By performing feature merging, the most relevant features can be extracted and selected, reducing the dimensionality of features, lowering the complexity of the model, avoiding the risk of overfitting, and improving the training efficiency and prediction performance of the model. Next, the merged features are input into the structure search module, which performs continuous encoding on the merged features to uncover potential correlations between feature values ​​and output feature data. This captures more action details, facilitating subsequent human behavior recognition and improving its accuracy. Furthermore, by constructing and combining new features, more discriminative and expressive features can be obtained. The output module then uses the feature data for behavior prediction, obtaining the existence probability vector and behavior probability vector corresponding to the predicted target. This determines whether the behavior corresponding to the predicted target exists and its category, improving the accuracy of specific behavior recognition. Finally, the action category of the human action is determined based on the existence probability vector and behavior probability vector, further enhancing the accuracy of human action recognition.

[0045] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description and the accompanying drawings. Attached Figure Description

[0046] The accompanying drawings are used to provide a further understanding of the technical solutions of this application and form part of the specification. They are used together with the examples of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0047] Figure 1 This is a flowchart of a specific method for the human behavior recognition method based on millimeter wave integration provided in the embodiments of this application;

[0048] Figure 2 This is a flowchart illustrating the specific process of feature merging of a first millimeter-wave signal and a second millimeter-wave signal using a preset neural network model, as provided in this application embodiment.

[0049] Figure 3This is a flowchart illustrating the specific process of the structure search module performing continuous encoding on merged features, as provided in this embodiment of the application.

[0050] Figure 4 This is a flowchart illustrating a human behavior recognition method based on millimeter-wave integration, provided in another embodiment of this application.

[0051] Figure 5 This is a flowchart illustrating the specific process of predicting behavior from feature data using an output module, as provided in an embodiment of this application.

[0052] Figure 6 This is a flowchart illustrating the process of determining the action category of a human action based on an existence probability vector and an action probability vector, as provided in an embodiment of this application.

[0053] Figure 7 This is a flowchart illustrating a human behavior recognition method based on millimeter-wave integration, provided in another embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the structure of the human behavior recognition system based on millimeter wave integration provided in the embodiments of this application;

[0055] Figure 9 This is a schematic diagram of the overall framework provided in one example of this application;

[0056] Figure 10 This is a schematic diagram of the input module provided in one example of this application;

[0057] Figure 11 This is a schematic diagram of the structure search module provided in an example of this application;

[0058] Figure 12 This is a schematic diagram of the structure search module provided in an example of this application;

[0059] Figure 13 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] The human behavior recognition method based on millimeter-wave integration provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application implementing the above method, but is not limited to the above forms.

[0064] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0065] Multi-target human behavior recognition technology based on millimeter-wave data can capture reflected waves using millimeter-wave radar sensors to analyze human movements, such as determining whether a person has fallen or is standing. This technology leverages the high penetration of millimeter waves, enabling effective monitoring even in environments with obstructed visibility and minimal interference from changes in lighting and weather. By providing radar point cloud data rather than clear images, it helps protect personal privacy and is primarily used in security monitoring, elderly care, and smart home systems.

[0066] However, the point cloud data generated by low-cost millimeter-wave radar systems is often sparse and uneven, which poses a challenge to training fine-grained and accurate activity classifiers. Due to the sparsity of the data, it may be impossible to capture all the necessary motion details, thus affecting the accuracy of recognition.

[0067] To address the aforementioned issues, this application provides a method, system, electronic device, and storage medium for human behavior recognition based on millimeter-wave integration. First, it acquires first millimeter-wave signals from multiple radar sensors, obtaining millimeter-wave signals at different angles. This facilitates subsequent analysis of human movement using captured reflected waves. Then, it transposes the first millimeter-wave signals to change their input dimension, resulting in a second millimeter-wave signal. This transforms the first millimeter-wave signal into another input dimension, yielding two millimeter-wave signals with different dimensions. A pre-defined neural network model then merges the features of the first and second millimeter-wave signals, encoding features from different dimensions into feature data to obtain merged features. Feature merging allows for the extraction and selection of the most relevant features, reducing feature dimensionality, lowering model complexity, avoiding overfitting risks, and improving model training efficiency and prediction performance. Next, the merged features are input into the structure search module, which performs continuous encoding on the merged features to uncover potential correlations between feature values ​​and output feature data. This captures more action details, facilitating subsequent human behavior recognition and improving its accuracy. Furthermore, by constructing and combining new features, more discriminative and expressive features can be obtained. The output module then uses the feature data for behavior prediction, obtaining the existence probability vector and behavior probability vector corresponding to the predicted target. This determines whether the behavior corresponding to the predicted target exists and its category, improving the accuracy of specific behavior recognition. Finally, the action category of the human action is determined based on the existence probability vector and behavior probability vector, further enhancing the accuracy of human action recognition.

[0068] Reference Figure 1 , Figure 1This is a flowchart illustrating a specific method of the millimeter-wave integrated human behavior recognition method provided in this application embodiment. In some embodiments, the millimeter-wave integrated human behavior recognition method includes, but is not limited to, steps S101 to S106.

[0069] Step S101: Acquire the first millimeter-wave signal from multiple radar sensors. The first millimeter-wave signal is obtained by capturing the reflected wave from human body movement by the radar sensors.

[0070] In step S101 of some embodiments, the radar sensor captures the reflected waves reflected by human body movements. In this embodiment, the first millimeter wave signals of multiple radar sensors are first millimeter wave signals at different angles, so as to achieve comprehensive capture of millimeter wave signals in different directions, which is convenient for subsequent analysis of human body movements through the first millimeter wave signals, such as determining whether the human body has fallen or stood up.

[0071] Understandably, millimeter waves' high penetration allows for effective monitoring in environments with obstructed line of sight, and are less affected by changes in lighting and weather. Furthermore, by providing radar point cloud data rather than clear images, it helps protect individual privacy.

[0072] Step S102: Transpose the first millimeter-wave signal to obtain the second millimeter-wave signal.

[0073] In step S102 of some embodiments, after acquiring the first millimeter-wave signal, the first millimeter-wave signal is transposed to transform it into another input dimension to obtain the second millimeter-wave signal. Specifically, in this embodiment, the millimeter-wave signal is transformed into another input dimension through a matrix transpose operation. The matrix transpose operation facilitates the subsequent model to learn the features of the data from different angles and dimensions, which helps to improve the model's generalization ability for different actions and enable it to maintain a high recognition accuracy when facing new and unseen actions.

[0074] Step S103: The first millimeter wave signal and the second millimeter wave signal are combined using a preset neural network model to obtain the combined features. The neural network model includes a structure search module and an output module.

[0075] In step S103 of some embodiments, the first millimeter wave signal and the second millimeter wave signal are feature-merged by a preset neural network model. By constructing and combining new features, more discriminative and expressive merged features can be obtained. These new features can often reveal hidden patterns and correlations in the data, thereby improving the predictive performance of the model and avoiding the inability to capture all necessary action details due to data sparsity.

[0076] Step S104: Input the merged features into the structure search module so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions.

[0077] In step S104 of some embodiments, the merged features are input into the structure search module so that the structure search module performs continuous encoding operations on the merged features, thereby achieving efficient encoding of the features, which can uncover the potential correlation between feature values ​​and output the feature vectors of all prediction targets to obtain feature data, which is convenient for subsequent behavior prediction.

[0078] Step S105: The output module performs behavior prediction on the feature data to obtain the existence probability vector and behavior probability vector corresponding to the prediction target.

[0079] In step S105 of some embodiments, the output module performs behavior prediction on the feature data to obtain the existence probability vector and behavior probability vector corresponding to the prediction target. The existence probability vector and behavior probability vector can provide the model with quantitative information about whether an action exists and the specific behavior category. Specifically, the existence probability vector can help identify whether a specific action exists, while the behavior probability vector can be used to classify the specific category of the action. The existence probability vector and behavior probability vector can improve the recognition accuracy of a specific action category or reduce the misidentification of some easily confused actions.

[0080] Step S106: Determine the action category of the human action based on the existence probability vector and the behavior probability vector.

[0081] In step S106 of some embodiments, the existence of the target human body and the behavior category are determined based on the existence probability vector and the behavior probability vector, thereby determining the action category of the human body action, achieving accurate recognition of human body action and improving the accuracy of human body action recognition.

[0082] Reference Figure 2 , Figure 2 This is a flowchart illustrating the specific process of feature merging of a first millimeter-wave signal and a second millimeter-wave signal using a preset neural network model, as provided in this application embodiment. In some embodiments, the method includes, but is not limited to, steps S201 to S202.

[0083] It should be noted that the neural network model includes an input module, and each input module includes a first convolutional layer and a second convolutional layer. The first convolutional layer is a 1*3 convolutional layer, which includes 1*3 sets of convolution operations, batch normalization operations, and ReLU activation operations. The second convolutional layer is a 1*1 convolutional layer, which includes 1*1 sets of convolution operations, batch normalization operations, and ReLU activation operations.

[0084] Step S201: Input the first millimeter-wave signal and the second millimeter-wave signal into a preset neural network model, so that the first convolutional layer performs feature encoding operation on the first millimeter-wave signal and outputs the first feature data, and the second convolutional layer performs feature encoding operation on the second millimeter-wave signal and outputs the second feature data.

[0085] Step S202: Merge the first feature data and the second feature data to obtain merged features.

[0086] In steps S201 to S202 of some embodiments, during the feature merging process of the first millimeter-wave signal and the second millimeter-wave signal using a preset neural network model, firstly, the first millimeter-wave signal and the second millimeter-wave signal are input into the preset neural network model, so that the first convolutional layer performs feature encoding operation on the first millimeter-wave signal and outputs first feature data, and the second convolutional layer performs feature encoding operation on the second millimeter-wave signal and outputs second feature data. The first and second convolutional layers mainly perform feature encoding on the input data at different dimensions with minimal computational cost and generate feature data. Then, the first feature data and the second feature data are merged to obtain merged features. By constructing and combining new features, more discriminative and expressive features can be obtained. These new features can often reveal hidden patterns and correlations in the data, thereby improving the predictive performance of the model and avoiding the inability to capture all necessary action details due to data sparsity.

[0087] In some embodiments, the present application can establish communication with other input modules by minimizing the mean squared error (MSE) loss function, and perform mutual supervision learning on each other's output features.

[0088] Specifically, the mean squared error loss function is expressed as follows:

[0089]

[0090] Among them, in the loss function In the middle, F i and F j Let represent the output feature maps of the i-th and j-th input modules, respectively, and k represent the total number of input modules.

[0091] Reference Figure 3 , Figure 3 This is a flowchart illustrating the specific process of the structure search module performing continuous encoding on merged features, as provided in this application embodiment. In some embodiments, the method includes, but is not limited to, steps S301 to S305.

[0092] It should be noted that the structure search module includes fully connected layers and a third convolutional layer. In this embodiment, the structure search module includes five fully connected layers.

[0093] Step S301: The merged features are aggregated using the structure search module to obtain the first intermediate feature.

[0094] Step S302: Input the intermediate features into the fully connected layer so that the fully connected layer encodes the intermediate features based on the preset search coefficients and outputs the encoded features.

[0095] Step S303: Transpose the encoded features to obtain the second intermediate features.

[0096] Step S304: Input the second intermediate feature into the third convolutional layer for feature capture and output convolutional feature information, wherein the convolutional feature information includes the total number of targets predicted.

[0097] Step S305: Transpose the convolutional feature information to output the feature data.

[0098] In steps S301 to S305 of some embodiments, during the continuous encoding operation of the merged features by the structure search module, firstly, the merged features are aggregated by the structure search module to obtain a first intermediate feature. Specifically, the row features are averaged and summed through the aggregation operation to obtain the first intermediate feature. Then, the intermediate feature is input into the fully connected layer so that the fully connected layer encodes the intermediate feature based on a preset search coefficient. Specifically, the fully connected layer performs a structured full connection operation, a batch normalization operation, and a ReLU activation operation on the intermediate feature, and outputs the encoded feature. In this embodiment, the structured full connection operation adds a search coefficient compared to the ordinary full connection operation, which is used for... The output connections of each neuron are weighted to measure the importance of a neuron, thereby enabling the discovery of potential correlations between feature values ​​and achieving efficient feature encoding. Then, the encoded features are transposed to obtain second intermediate features, realizing the dimensionality transformation of the encoded features. The second intermediate features are then input into the third convolutional layer for feature capture, outputting convolutional feature information, which includes the total number of predicted targets, facilitating subsequent behavior recognition based on convolutional feature information. Finally, the convolutional feature information is transposed to output feature data, thereby enabling the discovery of potential correlations between feature values ​​and outputting feature vectors of all predicted targets, obtaining feature data for subsequent behavior prediction.

[0099] Reference Figure 4 , Figure 4This is a flowchart illustrating a human behavior recognition method based on millimeter-wave integration, provided in another embodiment of this application. In some embodiments, the method includes, but is not limited to, steps S401 to S402.

[0100] Step S401: Apply regularization constraints to the search coefficients.

[0101] Step S402: Weight the output connections of neurons in each fully connected layer using search coefficients after regularization constraints.

[0102] In steps S401 to S402 of some embodiments, the present application embodiments will also apply regularization constraints to the search coefficients to make their values ​​sparse, thereby capturing more feature details. Then, the output connections of each fully connected layer neuron are weighted by the regularized search coefficients to measure the importance of different neurons.

[0103] In some embodiments, after weighting the output connections of neurons in each fully connected layer by search coefficients after regularization constraints, neurons with smaller search coefficients indicate that their output connections contribute little to the entire network and can be directly pruned. Through this mechanism, embodiments of this application can obtain a more compact and concise network structure, effectively reducing the computational load of fully connected layers.

[0104] Reference Figure 5 , Figure 5 This is a flowchart illustrating the behavior prediction of feature data through an output module, as provided in an embodiment of this application. In some embodiments, the method includes, but is not limited to, steps S501 to S503.

[0105] It should be noted that the output module includes a search fully connected layer, and in this embodiment of the application, the output module includes two search fully connected layers.

[0106] Step S501: The feature data is separated by the output module to obtain the feature vector corresponding to the prediction target.

[0107] Step S502: For each feature vector, input the feature vector into the search fully connected layer for continuous encoding operation, and output the third intermediate feature.

[0108] Step S503: Perform a fully connected operation on the third intermediate feature through the output module to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

[0109] In steps S501 to S503 of some embodiments, the output module is mainly responsible for receiving feature data from the search module, globally encoding the feature vector of each predicted target, and generating the final prediction result. During the behavior prediction process using the feature data through the output module, feature separation is first performed on the feature data to obtain the feature vector of a single predicted target, i.e., the feature vector corresponding to the predicted target. For each feature vector, the feature vector is input into a search fully connected layer for continuous encoding. Specifically, two search fully connected layers are used to continuously encode the feature vector, obtaining intermediate features of the same dimension, and outputting a third intermediate feature. Finally, the output module performs a fully connected operation on the third intermediate feature to generate an existence probability vector and a behavior probability vector. The existence probability vector and behavior probability vector provide the model with quantitative information about whether an action exists and the specific behavior category. Specifically, the existence probability vector helps identify whether a specific action exists, while the behavior probability vector can be used to classify the specific category of the action. The existence probability vector and behavior probability vector can improve the accuracy of identifying specific action categories or reduce misidentification of certain easily confused actions.

[0110] Reference Figure 6 , Figure 6 This is a flowchart illustrating the specific process of determining the action category of a human action based on an existence probability vector and a behavior probability vector, as provided in an embodiment of this application. In some embodiments, the method includes, but is not limited to, steps S601 to S603.

[0111] Step S601: Perform a summation and averaging operation on all existence probability vectors and behavior probability vectors to obtain a probability vector.

[0112] Step S602: Compare all probability vectors to select the target probability vector with the largest probability.

[0113] Step S603: Determine the action category of the human body based on the target probability vector.

[0114] In steps S601 to S603 of some embodiments, in the process of determining the action category of human body action based on the existence probability vector and the behavior probability vector, the embodiments of this application first perform a summation and averaging operation on all existence probability vectors and behavior probability vectors to obtain multiple probability vectors, then compare all probability vectors to filter out the target probability vector with the largest value, and then determine whether the target human body exists and which behavior category it belongs to based on the existence probability vector and behavior probability vector corresponding to the target probability vector, thereby achieving accurate recognition of human body action and improving the accuracy of human body action recognition.

[0115] Reference Figure 7 , Figure 7This is a flowchart illustrating a human behavior recognition method based on millimeter-wave integration, provided in another embodiment of this application. In some embodiments, the method includes, but is not limited to, steps S701 to S702.

[0116] Step S701: Perform mutual supervision learning on the structure search module using a preset mean squared error loss function.

[0117] In step S701 of some embodiments, the present application embodiments will also perform mutual supervision learning on the structure search modules through a preset mean squared error loss function. Specifically, during the training process, each structure search module performs mutual supervision learning with other search modules by minimizing the mean squared error loss function. It can learn from each other through the intermediate features or probability vectors of different modules and jointly improve the optimal solution of the network.

[0118] Understandably, the mean squared error loss function is expressed as follows:

[0119]

[0120] In the loss function In the middle, F u and F v Let represent the output feature maps of the u-th and v-th structure search modules, respectively, and let l represent the total number of structure search modules.

[0121] Step S702: Train the output module to perform human behavior training using a preset cross-entropy loss function, and perform mutual supervision learning on the output module using a preset divergence distance loss function.

[0122] In step S702 of some embodiments, the embodiments of this application will also train the output module on human behavior using a preset cross-entropy loss function, thereby improving the behavior prediction accuracy of the neural network model, and perform mutual supervision learning on the output module using a preset divergence distance loss function. By performing mutual supervision learning on the output module, the behavior probability distribution of other output modules is learned, thereby improving the optimal solution of the network.

[0123] Specifically, the cross-entropy loss function in this application embodiment includes a first cross-entropy loss function and a second cross-entropy loss function. The first cross-entropy loss function is used to learn the target existence task, making the existence probability vector generated by the output module more accurate. The second cross-entropy loss function is used to learn the behavior recognition task, thereby making the behavior probability vector of the output module more accurate.

[0124] Understandably, the first cross-entropy loss function is expressed as follows:

[0125]

[0126] Among them, yi and These represent the true label and the predicted probability of whether the target exists, respectively.

[0127] The second cross-entropy loss function is expressed as follows:

[0128]

[0129] Among them, b i and These represent the true label and predicted probability of the target's current behavior, respectively. B represents the total number of behavior categories, with common human behavior categories including falling, standing, and lying still.

[0130] The divergence distance loss function is expressed as follows:

[0131]

[0132] in, and Let K and H represent the predicted probabilities of the i-th behavior of the k-th and h-th output modules, respectively, where K represents the total number of output modules.

[0133] Please see Figure 8 This application also provides a human behavior recognition system based on millimeter-wave integration, which can implement the above-mentioned human behavior recognition method based on millimeter-wave integration. The system includes:

[0134] The data acquisition module 801 is used to acquire the first millimeter-wave signals from multiple radar sensors. The first millimeter-wave signals are captured by the radar sensors from the reflected waves reflected by human body movements.

[0135] The signal transpose module 802 is used to transpose the first millimeter-wave signal to obtain the second millimeter-wave signal.

[0136] The feature merging module 803 is used to merge the features of the first millimeter wave signal and the second millimeter wave signal through a preset neural network model to obtain merged features. The neural network model includes a structure search module and an output module.

[0137] The continuous encoding module 804 is used to input the merged features into the structure search module so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions.

[0138] The behavior prediction module 805 is used to perform behavior prediction on feature data through the output module to obtain the existence probability vector and behavior probability vector corresponding to the prediction target.

[0139] The category determination module 806 is used to determine the action category of human actions based on the existence probability vector and the behavior probability vector.

[0140] To more clearly illustrate the human behavior recognition method, system, and electronic device based on millimeter wave integration of the embodiments of this application, specific examples are provided below.

[0141] Example 1:

[0142] Please see Figure 9 , Figure 9 This is a schematic diagram of the overall framework provided in one example of this application.

[0143] like Figure 9 As shown in the embodiment, this application can receive millimeter-wave signals from different angles. Here, it is assumed that each millimeter-wave signal has a dimension of 128*5, where 128 represents the number of radar point clouds, and 5 represents the number of features, representing the spatial coordinates (x, y, z), signal transmission rate, and reflection intensity of each point, respectively. The millimeter-wave signal from each angle sequentially passes through the input module, structure search module, and output module, ultimately generating N target existence probability vectors (1*2) and behavior probability vectors (1*B), where B represents the total number of behavior categories. During training, intermediate features or probability vectors from different modules mutually supervise each other, jointly improving the optimal solution of the network. Finally, a final prediction result is obtained by voting on all output results.

[0144] The following section provides a detailed explanation of the millimeter-wave integrated human behavior recognition method in this example, using different modules as examples.

[0145] Please see Figure 10 , Figure 10 This is a schematic diagram of the input module provided in one example of this application.

[0146] In some embodiments, the input module employs a dual-input structure, transforming the millimeter-wave signal into another input dimension: 5*128 through matrix transpose. The two different forms of millimeter-wave data are respectively processed by a 1*3 convolutional layer (containing 1*3 sets of convolution operations, batch normalization, and ReLU activation) and a 1*1 convolutional layer (containing 1*1 convolution operations, batch normalization, and ReLU activation). These layers primarily encode features at different dimensional levels of the input data with minimal computational cost, generating 128*256 and 5*256 feature data. Finally, a merging operation yields 133*256 output features. During training, each input module establishes communication with other input modules by minimizing the mean squared error loss function, mutually supervising each other's output features.

[0147] Please see Figure 11 , Figure 11This is a schematic diagram of the structure search module provided in an example of this application.

[0148] In some embodiments, the structure search module serves as the main structure of the model, primarily responsible for continuously encoding the feature data of the input module, mining the potential correlations between feature values, and outputting feature vectors for all predicted targets. Within the structure search module, we emphasize the introduction of a fully connected layer based on a structured search mechanism. This fully connected layer mainly includes structured fully connected operations, batch normalization operations, and ReLU activation operations. The structured fully connected operation adds a search coefficient compared to ordinary fully connected operations, used to weight all output connections of each neuron to measure the importance of a neuron.

[0149] In network training, embodiments of this application apply L1 regularization constraints to all search coefficients to induce sparsity in their values. For example... Figure 11 As shown, W3 = [0.8, 0.4, 0.001, 0.09] represents the search coefficients of all neurons in the third fully connected layer. The search coefficients of the third neuron are close to zero, indicating that its output connection contributes very little to the entire network and can be directly pruned. Through this mechanism, we can obtain a more compact and concise network structure, effectively reducing the computational cost of fully connected layers. In the structure search module, the input features are first averaged through an aggregation operation to obtain 1*256 intermediate features. These features are then efficiently encoded through five fully connected layers based on the structure search mechanism. Finally, an N*256 output feature is generated through a transpose operation and a 1*1 convolutional layer, where N represents the total number of predicted targets.

[0150] During training, each structure search module learns from other search modules by minimizing the mean squared error loss function.

[0151] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure search module provided in an example of this application.

[0152] In some embodiments, the output module is mainly responsible for receiving feature data from the search module, globally encoding the feature vector of each predicted target, and generating the final prediction result. In this module, the target feature map (N*256) first obtains the feature vector (1*256) of a single predicted target through a separation operation, then performs continuous encoding through two fully connected layers based on a structured search mechanism to obtain intermediate features of the same dimension, and finally generates an existence probability vector (1*2) and a behavior probability vector (1*B) through two fully connected operations respectively.

[0153] During training, the model uses a cross-entropy loss function to learn the target presence task. The model also employs the cross-entropy loss function for the behavior recognition task. In this module, in addition to learning the target presence and behavior recognition tasks, this embodiment also introduces a loss function based on KL (Kullback-Leibler Divergence) divergence distance to learn the behavior probability distributions of other output modules.

[0154] The method in this application embodiment can not only effectively improve behavior recognition performance by utilizing multi-angle millimeter wave data, but is also designed as a lightweight neural network model, which is more friendly to edge device deployment.

[0155] The human behavior recognition system based on millimeter wave integration in this application embodiment is used to execute the human behavior recognition method based on millimeter wave integration in the above embodiments. Its specific processing procedure is the same as that of the human behavior recognition method based on millimeter wave integration in the above embodiments, and will not be described in detail here.

[0156] This application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, is used by the processor to execute the millimeter-wave integrated human behavior recognition method in the above embodiments of this application.

[0157] Please see Figure 13 , Figure 13 The illustration shows the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0158] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0159] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and called and executed by the processor 901 to implement the millimeter-wave integrated human behavior recognition method of the embodiments of this application.

[0160] The 903 input / output interface is used to implement information input and output.

[0161] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0162] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0163] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0164] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described millimeter-wave integrated human behavior recognition method.

[0165] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0166] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0167] It will be understood by those skilled in the art that Figure 1-13 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0168] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0169] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0170] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0171] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0172] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0173] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A human behavior recognition method based on millimeter-wave integration, characterized in that, The method includes: The first millimeter-wave signal from multiple radar sensors is acquired, and the first millimeter-wave signal is obtained by capturing the reflected wave reflected by the radar sensor in response to human movement; The first millimeter-wave signal is transposed to obtain the second millimeter-wave signal; The first millimeter-wave signal and the second millimeter-wave signal are combined using a preset neural network model to obtain combined features. The neural network model includes a structure search module and an output module. The merged features are input into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions; The output module performs behavior prediction on the feature data to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target. The action category of the human action is determined based on the existence probability vector and the behavior probability vector; The structure search module includes a fully connected layer and a third convolutional layer; the step of inputting the merged features into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, includes: The structure search module performs feature aggregation on the merged features to obtain a first intermediate feature; wherein, the feature aggregation refers to averaging and summing the row features through aggregation operations; The intermediate features are input into the fully connected layer, so that the fully connected layer encodes the intermediate features based on preset search coefficients and outputs encoded features; wherein, the fully connected layer performs structured full connection operation, batch normalization operation and ReLU activation operation on the first intermediate features and outputs the encoded features, and the structured full connection operation is used to weight all output connections of each neuron by the search coefficients; The encoded feature is transposed to obtain the second intermediate feature; The second intermediate feature is input into the third convolutional layer for feature capture, and convolutional feature information is output, wherein the convolutional feature information includes the total number of targets in the predicted target; The convolutional feature information is transposed to output feature data; The method further includes: Apply regularization constraints to the search coefficients; The output connections of each neuron in the fully connected layer are weighted by search coefficients after regularization constraints.

2. The human behavior recognition method based on millimeter-wave integration according to claim 1, characterized in that, The neural network model includes a first convolutional layer and a second convolutional layer; the feature merging of the first millimeter-wave signal and the second millimeter-wave signal using the preset neural network model to obtain merged features includes: The first millimeter-wave signal and the second millimeter-wave signal are input into a preset neural network model, so that the first convolutional layer performs feature encoding operation on the first millimeter-wave signal and outputs first feature data, and the second convolutional layer performs feature encoding operation on the second millimeter-wave signal and outputs second feature data. The first feature data and the second feature data are merged to obtain merged features.

3. The human behavior recognition method based on millimeter-wave integration according to claim 1, characterized in that, The output module includes a search fully connected layer; the step of performing behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target includes: The output module performs feature separation on the feature data to obtain the feature vector corresponding to the prediction target; For each feature vector, the feature vector is input into the search fully connected layer for continuous encoding operation, and the third intermediate feature is output. The output module performs a fully connected operation on the third intermediate feature to output an existence probability vector and a behavior probability vector, wherein the behavior probability vector is used to represent the total number of behavior categories.

4. The human behavior recognition method based on millimeter-wave integration according to claim 1, characterized in that, Determining the action category of the human action based on the existence probability vector and the behavior probability vector includes: The probability vector is obtained by summing and averaging all the existence probability vectors and the behavior probability vectors. Compare all the probability vectors to select the target probability vector with the largest value; The action category of the human body action is determined based on the target probability vector.

5. The human behavior recognition method based on millimeter-wave integration according to claim 1, characterized in that, The method further includes: The structure search module is subjected to mutual supervision learning through a preset mean square error loss function; The output module is trained on human behavior using a preset cross-entropy loss function, and the output module is subjected to mutual supervision learning using a preset divergence distance loss function.

6. A human behavior recognition system based on millimeter-wave integration, applied to the human behavior recognition method based on millimeter-wave integration as described in any one of claims 1-5, characterized in that, The system includes: The data acquisition module is used to acquire the first millimeter-wave signals from multiple radar sensors, which are obtained by capturing the reflected waves from human body movements by the radar sensors. The signal transpose module is used to transpose the first millimeter-wave signal to obtain the second millimeter-wave signal; The feature merging module is used to merge the features of the first millimeter-wave signal and the second millimeter-wave signal through a preset neural network model to obtain merged features. The neural network model includes a structure search module and an output module. A continuous encoding module is used to input the merged features into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, wherein the feature data includes multiple prediction targets corresponding to human actions; The behavior prediction module is used to perform behavior prediction on the feature data through the output module to obtain an existence probability vector and a behavior probability vector corresponding to the prediction target. A category determination module is used to determine the action category of the human action based on the existence probability vector and the behavior probability vector; The structure search module includes a fully connected layer and a third convolutional layer; the step of inputting the merged features into the structure search module, so that the structure search module performs continuous encoding operations on the merged features and outputs feature data, includes: The structure search module performs feature aggregation on the merged features to obtain a first intermediate feature; wherein, the feature aggregation refers to averaging and summing the row features through aggregation operations; The intermediate features are input into the fully connected layer, so that the fully connected layer encodes the intermediate features based on preset search coefficients and outputs encoded features; wherein, the fully connected layer performs structured full connection operation, batch normalization operation and ReLU activation operation on the first intermediate features and outputs the encoded features, and the structured full connection operation is used to weight all output connections of each neuron by the search coefficients; The encoded feature is transposed to obtain the second intermediate feature; The second intermediate feature is input into the third convolutional layer for feature capture, and convolutional feature information is output, wherein the convolutional feature information includes the total number of targets in the predicted target; The convolutional feature information is transposed to output feature data.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the human behavior recognition method based on millimeter wave integration as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the millimeter-wave integrated human behavior recognition method as described in any one of claims 1 to 5.

Images