Gait emotion recognition method based on neural network architecture search

Abstract

The application discloses a gait emotion recognition method based on neural network architecture search and belongs to the technical field of gait emotion recognition. The method comprises the following steps: acquiring a three-dimensional skeleton gait sequence and representing the three-dimensional skeleton gait sequence as a gait skeleton graph; constructing a search space containing seven candidate operations, constructing a directed acyclic graph through a sequential greedy search strategy and generating a module structure; adopting TCN noise disturbance optimization of the module structure, introducing a skip connection operation and optimizing search capability through random sampling of a feature channel; based on the optimized module structure, constructing a graph convolution network for gait emotion recognition, inputting the gait skeleton graph and outputting an emotion classification result. Through the sequential greedy search strategy, the application automatically generates a network topology structure suitable for gait data characteristics, reduces the degree of human participation, effectively improves the diversity and task adaptability of the model architecture, improves the emotion recognition performance through optimization of candidate operation selection and improves the model training convergence speed through an auxiliary skip path.

Description

Technical Field

[0001] This invention relates to the field of gait emotion recognition technology, specifically a gait emotion recognition method based on neural network architecture search. Background Technology

[0002] In the field of artificial intelligence, affective computing technology, by constructing a human-computer emotional interaction channel, not only provides a quantitative assessment tool for early screening of mental disorders but also lays the technological foundation for the development of digital therapies. Existing emotion recognition systems face a significant modality selection dilemma: while methods based on facial micro-expressions and acoustic features have high interpretability, they suffer from the risk of biometric leakage and cross-cultural sensitivity defects; in contrast, gait, as a non-contact biological behavioral feature, offers the dual advantages of anonymity and cross-scene representation stability through its spatiotemporal dynamic characteristics.

[0003] In recent years, deep learning has demonstrated remarkable performance in emotion recognition tasks thanks to its advantage of automatic feature extraction. Some researchers have proposed an autoencoder-based method to classify perceived human emotions from gait data and use LSTM and MLP to extract gait emotion representation information. However, these methods treat skeleton-based gait data as pseudo-images or text information, ignoring joint topology, leading to the loss of important discriminative information.

[0004] Deep learning technology has demonstrated outstanding performance in emotion recognition due to its automatic feature extraction capabilities. Existing research has proposed frameworks based on autoencoders, extracting emotional representation features from gait data through Long Short-Term Memory (LSTM) networks and Multilayer Perceptrons (MLPs). However, these methods simplify skeletal sequence data into pseudo-image or textual information processing, ignoring the intrinsic correlation of joint topology, leading to a severe loss of discriminative features. Graph Convolutional Networks (GCNs), due to their advantage in handling non-Euclidean topology data, have become one of the mainstream methods for skeletal gait emotion recognition. Typical studies include classification architectures based on Spatiotemporal Graph Convolutional Networks (ST-GCNs) and emotion feature extraction models such as Global Graph Convolutional Shrinking Networks (G-GCSNs) that integrate global connections and shrinking blocks. However, due to the specificity of emotion recognition tasks in feature space distribution, directly transferring existing architectures will lead to significant performance degradation, while targeted model design requires a large amount of time and computational resources. As a core technology of Automated Machine Learning (AutoML), Neural Architecture Search (NAS) aims to automatically generate efficient network structures with minimal computational cost. Existing NAS methods mainly fall into three paradigms: reinforcement learning (RL), evolutionary algorithms (EA), and gradient optimization. RL and EA methods typically consume up to 2,000 GPU-days of computational resources, while differentiable architecture search significantly reduces computational complexity by continuously relaxing the search space and utilizing gradient descent to optimize validation set performance. However, existing NAS methods are primarily designed for image classification tasks and are difficult to directly apply to gait emotion recognition, a domain with unique data characteristics.

[0005] Therefore, how to construct an emotion recognition task adapted to gait, and automatically assemble the structure of the final model with less time and effort to improve the final emotion recognition performance, has become a key problem that this invention urgently needs to solve. Summary of the Invention

[0006] The purpose of this invention is to provide a gait emotion recognition method based on neural network architecture search, so as to overcome the bottleneck of manually designing model structures in gait-based emotion recognition tasks; enhance the spatiotemporal feature extraction capability and improve emotion recognition performance; improve search efficiency and robustness, thereby increasing the model training convergence speed; fill the gap of NAS in gait emotion recognition and overcome the compatibility barrier between non-Euclidean data structures and NAS methods.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] A gait emotion recognition method based on neural network architecture search is proposed, which includes the following steps:

[0009] Obtain the 3D skeleton gait sequence and represent each frame of the gait skeleton sequence as a gait skeleton diagram;

[0010] A search space containing 7 candidate operations is constructed. A directed acyclic graph is constructed and a module structure is generated through a sequential greedy search strategy. The module structure is optimized by TCN noise perturbation, and the search capability is optimized by random sampling of feature channels.

[0011] Based on the optimized module structure, a graph convolutional network for gait emotion recognition is constructed, which takes a gait skeleton graph as input and outputs emotion classification results.

[0012] As an optimization scheme of the gait emotion recognition method based on neural network architecture search in this invention, the seven candidate operations include temporal convolution operation, four spatial graph convolution operations, no operation, and jump operation.

[0013] As an optimization scheme of the gait emotion recognition method based on neural network architecture search of the present invention, the directed acyclic graph includes intermediate nodes for storing the spatiotemporal features of the skeleton graph sequence, edges for determining the information flow direction from the previous intermediate node to the next intermediate node, edge operations for processing the data transmitted by the edges, and operation weights for controlling the adaptability of different operations to the skeleton features.

[0014] As an optimization scheme of the gait emotion recognition method based on neural network architecture search in this invention, the sequential greedy strategy implements a phased optimization mechanism, iteratively optimizing the intermediate node connection, edge operation and operation weight in sequence, and making pruning decisions based on three indicators: candidate operation importance, selection determinism and stability.

[0015] As an optimization scheme of the gait emotion recognition method based on neural network architecture search of the present invention, the specific implementation process of pruning decision based on three indicators—importance of candidate operations, selection determinism, and stability—includes:

[0016] Calculate the importance of candidate operations on the edges based on their architectural parameters. The specific formula is as follows:

[0017] ;

[0018] ;

[0019] Where 'o' represents the candidate operation selected from the set of all candidate operations that do not contain a zero operation. The architecture parameters representing the operation weights of candidate operation o on the edges are used to measure the weights of candidate operation o. Let the set of all candidate operations on the edge be represented. Indicates zero operation. Represents the softmax function This indicates a candidate operation selected from the set of all candidate operations containing a zero operation; the zero operation indicates that the connection does not exist.

[0020] It's important to explain that the sum of the softmax values ​​corresponding to the non-zero operation parameters serves as a measure of the importance of candidate operations. A higher sum of non-zero operation weights on an edge indicates a more crucial role for that edge in transmitting information within the network. In the softmax function, the numerator is the exponentialized weight of the candidate operation o, and the denominator is the sum of the exponentialized weights of all candidate operations. The ratio of these two is the normalized weight of candidate operation o. The softmax function assigns weights within the range [0,1] to discrete candidate operations, facilitating subsequent differentiable calculations with the network weights of the model. Ultimately, the importance of a candidate operation is the sum of the normalized weights of all non-zero candidate operations.

[0021] Normalized weights for non-zero operations Treating it as a distribution, the calculation formula is as follows:

[0022] ;

[0023] By combining the normalized entropy complement of the operation distribution with entropy calculation, we obtain the candidate operation selection determinism, which measures the degree of concentration of non-zero operations on the edge. The calculation formula is as follows:

[0024] ;

[0025] in, This represents the maximum possible entropy when all non-zero operations are uniformly distributed;

[0026] It's important to explain that calculating the candidate operation selection determinism measures the concentration of non-zero operations on a given edge, reflecting the algorithm's confidence in selecting a specific operation for that edge. The core idea is: the more concentrated the distribution (i.e., the significantly higher the weight of a particular non-zero operation compared to others), the higher the selection determinism. The detailed process is as follows: For each non-zero operation on an edge, calculate its normalized probability distribution. Calculate the normalized entropy of the distribution. The lower the entropy value, the more concentrated the distribution. The selection determinism is defined as the complement of the normalized entropy. The denominator is the maximum possible entropy (when all non-zero operations are uniformly distributed). This process ensures that the selection determinism index ranges from [0,1].

[0027] The average stability of the selected candidate operations is calculated using histogram intersection. The formula is as follows:

[0028] ;

[0029] in, Indicates the number of iterations. This represents the total number of iterations in the search process, where K represents any number of iterations within the range [0, T]. Indicates the number of iterations. The normalized weights of the candidate operation o, This represents the normalized weight of candidate operation o when the number of iterations is T;

[0030] Based on the three indicators mentioned above, the scores of the candidate operations are calculated. The calculation formula is as follows:

[0031] ;

[0032] Norm() represents the maximum and minimum value normalization operation;

[0033] Candidate operations are sorted in descending order based on their scores, and the first candidate operation is selected as the deterministic operation.

[0034] When two deterministic operations have been selected for the intermediate node, the candidate operations represented by the other incoming edges will be pruned to generate a module structure.

[0035] As an optimization scheme of the gait emotion recognition method based on neural network architecture search of the present invention, the specific implementation process of the TCN noise perturbation optimization module structure includes:

[0036] TCN noise perturbation is used to intervene in the TCN operation to obtain the selection determinism of the optimized candidate operation. The intervention formula is as follows:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] in, This indicates the determinism of candidate operation selection after adding the TCN perturbation. Indicates adding The sigmoid function used as weights A function representing the weights of the regularization architecture parameters. Indicates the architecture parameters Influence function, Indicates the optimized Function output, This indicates the weight given to balancing architectural parameters and validation losses in determining the final operation. Indicates architecture parameters learning rate This represents the normalized weights of non-zero operations after adding the TCN perturbation. Indicates the architecture parameters The output after performing the softmax operation. Indicates the emotional characteristics of the input. This represents Gaussian noise with zero mean and small variance.

[0044] The selection determinism and stability of the optimized candidate operations are calculated, and pruning decisions are made based on the selection determinism of the optimized candidate operations to obtain the optimized module structure.

[0045] As an optimization scheme of the gait emotion recognition method based on neural network architecture search of the present invention, the specific implementation process of introducing jump connection operations and optimizing search capabilities through random sampling of feature channels includes:

[0046] An auxiliary jump connect operation is introduced, which includes TGCN and identity mapping, to modify the candidate search space. The modification formula is as follows:

[0047] ;

[0048] in, This represents the modified candidate search space. These are the weight parameters of the auxiliary connections;

[0049] The output of a portion of the feature channels for each node is calculated using a sampling mechanism:

[0050] ;

[0051] in, Indicates the number of candidate operations. This indicates the probability of the selected channel. and represents the sampled channel and the unsampled channel, respectively, and k represents the k-th candidate operation that is not a TCN.

[0052] As an optimization scheme of the gait emotion recognition method based on neural network architecture search of the present invention, it also includes selecting optimized candidate operations for extracting emotion features between different units, wherein the first and second input nodes of each unit correspond to the outputs of the first two units respectively, and the output node of the unit is composed of the feature maps of all its intermediate nodes through depth concatenation.

[0053] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: The gait emotion recognition method based on neural network architecture search provided by this invention includes acquiring a three-dimensional skeleton gait sequence and representing each frame of the gait skeleton sequence as a gait skeleton graph; constructing a search space containing 7 candidate operations, constructing a directed acyclic graph and generating a module structure through a sequential greedy search strategy; optimizing the module structure using TCN noise perturbation and optimizing the search capability through random sampling of feature channels; based on the optimized module structure, constructing a graph convolutional network for gait emotion recognition, inputting the gait skeleton graph, and outputting the emotion classification result. This invention automatically generates a network topology structure adapted to the characteristics of gait data through a sequential greedy search strategy, effectively improving the diversity and task adaptability of the model architecture while reducing human intervention; improving emotion recognition performance by optimizing candidate operation selection; and improving the model training convergence speed by assisting in skipping paths. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the steps of the gait emotion recognition method based on neural network architecture search of the present invention;

[0056] Figure 2 This is a schematic diagram of the model architecture of the gait emotion recognition method based on neural network architecture search of the present invention. Detailed Implementation

[0057] To make the above-mentioned objects, features, and advantages of the present invention more readily understood, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0058] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0059] The gait emotion recognition method based on neural network architecture search proposed in this invention mainly comprises two stages: a search stage and a validation stage. In the search stage, the gait emotion recognition method designs a specific search space for skeleton data and proposes several search strategies to accelerate the automatic model design process for skeleton-based emotion recognition tasks. In the validation stage, the gait emotion recognition method utilizes the optimized modules searched in the previous stage to construct the final model structure and performs emotion classification on the skeleton-based data for validation.

[0060] Please see Figure 1 In this first embodiment: a gait emotion recognition method based on neural network architecture search is provided, which includes the following steps:

[0061] A gait emotion recognition method based on neural network architecture search includes the following steps:

[0062] Obtain the 3D skeleton gait sequence and represent each frame of the gait skeleton sequence as a gait skeleton diagram;

[0063] A search space containing 7 candidate operations is constructed. A directed acyclic graph is constructed and a module structure is generated through a sequential greedy search strategy. The module structure is optimized by TCN noise perturbation, and the search capability is optimized by random sampling of feature channels.

[0064] Based on the optimized module structure, a graph convolutional network for gait emotion recognition is constructed, which takes a gait skeleton graph as input and outputs emotion classification results.

[0065] Specifically, the seven candidate operations include temporal convolution operations, four spatial graph convolution operations, no operation, and jump operations.

[0066] Specifically, the directed acyclic graph includes intermediate nodes for storing the spatiotemporal features of the skeleton graph sequence, edges for determining the direction of information flow from the previous intermediate node to the next intermediate node, edge operations for processing the data transmitted by the edges, and operation weights for controlling the adaptability of different operations to the skeleton features.

[0067] Specifically, the sequential greedy strategy implements a phased optimization mechanism, iteratively optimizing the intermediate node connections, edge operations, and operation weights in sequence, and making pruning decisions based on three indicators: the importance of candidate operations, selection determinism, and stability.

[0068] To enhance the consistency of performance between the derived architectures in the search and validation phases, this invention proposes a sequential greedy strategy. This strategy aims to effectively reduce the deviation between structural design and weight distribution in gait emotion recognition methods based on neural network architecture search, thereby improving the stability and generalization ability of architecture transfer. Specifically, this method divides the overall optimization problem into several sub-problems and solves each sub-problem sequentially using a greedy algorithm, gradually constructing a high-performance neural architecture. To ensure that the selected modules have optimal performance, this invention systematically analyzes the candidate operation selection process from three aspects, making pruning decisions based on three indicators: candidate operation importance, selection determinism, and stability.

[0069] The importance of candidate operations on an edge is calculated based on the architectural parameters of the candidate operations, using the following formula:

[0070] ;

[0071] ;

[0072] Where 'o' represents a candidate operation selected from the set of all candidate operations that do not contain a zero operation. The architecture parameters representing the operation weights of candidate operation o on the edges are used to measure the weights of candidate operation o. Let the set of all candidate operations on the edge be represented. Indicates zero operation. Represents the softmax function This indicates a candidate operation selected from the set of all candidate operations containing a zero operation; the zero operation indicates that the connection does not exist.

[0073] Normalized weights for non-zero operations Treating it as a distribution, the calculation formula is as follows:

[0074] ;

[0075] Entropy is often used to measure the uncertainty of a distribution. By combining entropy with the normalized entropy complement of the operation distribution, we obtain the candidate operation selection determinism, which measures the degree of concentration of non-zero operations on an edge. The calculation formula is as follows:

[0076] ;

[0077] in, This represents the maximum possible entropy when all non-zero operations are uniformly distributed;

[0078] Histogram intersection is a commonly used method to detect changes in distribution. It calculates the average stability of the selected candidate operations by using histogram intersection to obtain the stability of the chosen operations. The formula is as follows:

[0079] ;

[0080] in, Indicates the number of iterations. This represents the total number of iterations in the search process, where K represents any number of iterations within the range [0, T]. Indicates the number of iterations. The normalized weights of the candidate operation o, This represents the normalized weight of candidate operation o when the number of iterations is T;

[0081] Based on the three indicators mentioned above, the scores of the candidate operations are calculated. The calculation formula is as follows:

[0082] ;

[0083] Norm() represents the maximum and minimum value normalization operation;

[0084] Candidate operations are sorted in descending order based on their scores, and the first candidate operation is selected as the deterministic operation.

[0085] When two deterministic operations have been selected for the intermediate node, the candidate operations represented by other incoming edges will be pruned to generate a module structure. This feature helps to alleviate the performance gap between the module structures in the search phase and the verification phase.

[0086] Specifically, the implementation process of the TCN noise perturbation optimization module structure includes:

[0087] Motion information corresponding to each frame is crucial for recognizing pedestrian walking. In emotion recognition tasks, extracting spatial feature information becomes a core issue. However, TCN (Tracking Channel Network) is more frequently used in some studies. This trend is considered an advantage of TCN during the search phase. The "enrichment phenomenon" of TCN leads to a large number of TCN operations in the module structure, thus limiting the final network's recognition performance. To address this issue, this invention proposes an optimization strategy for TCN or the search structure to limit the advantage of TCN operations. First, it directly intervenes in TCN operations, weakening their operational advantage without affecting feature distribution calculation.

[0088] ;

[0089] in, Indicates the architecture parameters of TCN operation. Yes The output after performing the softmax operation. Indicates the emotional characteristics of the input. This represents the added noise, which is Gaussian noise with zero mean and small variance. Therefore, and The calculation process is modified as follows:

[0090] ;

[0091] ;

[0092] From the regularization strategy employed, L2 regularization is applied to the architecture parameters, and its update method is as follows: .in, The learning rate represents the architecture parameter. This indicates normalization. However, for adaptive gradient optimization algorithms, L2 regularization has a relatively uniform effect on each element, which weakens its effectiveness to some extent. To improve the robustness and generalization ability of the proposed method, this method modifies the function... Modifications have been made to enhance... The expressive power of this function. The combination is used to indicate the importance of each operation and helps to filter out non-TCN operations.

[0093] ;

[0094] in, This represents the softmax function used to compute the corresponding operation. To indicate the corresponding time steps, we use... as independent variable A function is used to represent the overall effect of the modified regularization term. Similar to L2 regularization, a unified formula is ultimately obtained: Therefore, the above formula can be modified as follows:

[0095] ;

[0096] in, Indicates to and The influence function. For the function It needs to meet some conditions: First, Should not be subject to To mitigate the impact of ineffective regularization and optimization difficulties, we adopt... .

[0097] Therefore, to obtain the selection determinism of the optimized candidate operations, the intervention formula is as follows:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, This indicates the determinism of candidate operation selection after adding the TCN perturbation. Indicates adding The sigmoid function used as weights A function representing the weights of the regularization architecture parameters. Indicates the architecture parameters Influence function, Indicates the optimized Function output, This indicates the weight given to balancing architectural parameters and validation losses in determining the final operation. Indicates architecture parameters learning rate This represents the normalized weights of non-zero operations after adding the TCN perturbation. Indicates the architecture parameters The output after performing the softmax operation. Indicates the emotional characteristics of the input. This represents Gaussian noise with zero mean and small variance.

[0105] The selection determinism and stability of the optimized candidate operations are calculated, and pruning decisions are made based on the selection determinism of the optimized candidate operations to obtain the optimized module structure.

[0106] Specifically, the implementation process of introducing skip connection operations and optimizing search capabilities through random sampling of feature channels includes:

[0107] Because the proposed method focuses more on TCN or spatiotemporal GCN, skip connections are ignored during the search phase. Skip connection operations can improve the stability of the training process and maintain the robustness of the final network structure. Therefore, this invention introduces an auxiliary skip connection operation, including TGCN or Identity Operation, to modify the candidate search space, as shown in the following formula:

[0108] ;

[0109] in, This represents the modified candidate search space. These are the weight parameters of the auxiliary connections;

[0110] The expansion of the search space increases the search time, thus increasing computational resource consumption. To accelerate the search process, we use a sampling mechanism to compute only the output of a subset of feature channels for each node:

[0111] ;

[0112] in, Indicates the number of candidate operations. This indicates the probability of the selected channel. and represents the sampled channel and the unsampled channel, respectively, and k represents the k-th candidate operation that is not a TCN.

[0113] This sampling strategy significantly saves computational resources for gait emotion recognition methods based on neural network architecture search.

[0114] Specifically, it also includes selecting optimized candidate operations for extracting emotional features between different units. The first and second input nodes of each unit correspond to the outputs of the first two units, and the output node of a unit is composed of the feature maps of all its intermediate nodes through a depth-stitching method.

[0115] This invention proposes a gait emotion recognition method based on neural network architecture search. A schematic diagram of the model architecture of this method is shown below. Figure 2 As shown in the figure, the green area represents the search phase of the gait emotion recognition method based on neural network architecture search, which is mainly used to automatically search for the optimal module structure; the blue area corresponds to the verification phase, in which the optimal structure obtained by the search is used to build the final emotion recognition model and its performance is systematically verified.

[0116] This invention proposes a gait emotion recognition method based on neural network architecture search, which mainly comprises two stages: a search stage and a validation stage. In the search stage, the gait emotion recognition method based on neural network architecture search designs a specific search space for skeleton data and proposes several search strategies to accelerate the automatic model design process for skeleton-based emotion recognition tasks. In the validation stage, the gait emotion recognition method based on neural network architecture search utilizes the optimized modules found in the previous stage to construct the final model structure and validates it by classifying emotions from skeleton-based data.

[0117] Given a 3D skeleton gait sequence (Skeleton-based Data) ,in Indicates the first Frame, containing A body joint, Representing the three-dimensional coordinates of each joint This invention aims to automatically design an efficient encoder model based on a graph convolutional network (GCN) to process the input gait skeleton sequence. The mapping is performed as a discriminative feature representation to achieve accurate matching with corresponding sentiment tags. This invention maps each frame of the gait skeleton sequence... Represented as a gait skeleton diagram The data is then input into the proposed gait emotion recognition method based on neural network architecture search. This method consists of two phases: a search phase and a validation phase. In the search phase, i.e. Figure 2 The green section describes a GCN-based search space designed to explore the diversity of GCN architectures. This space contains seven common operations. Next, the search module constructs a Directed Acyclic Graph (DAG) using selected computational operations and processed sentiment features. Figure 2 Pink rectangle N in the middle green section With corresponding curves Here, edges represent optional operations, and pink rectangles represent extracted sentiment features. To efficiently select the optimal operation for each node, a gait sentiment recognition method based on neural network architecture search proposes a sequential greedy search strategy, while optimizing candidate operations to improve search accuracy and efficiency. Furthermore, considering the high computational cost of traditional GCN methods due to their dependence on the entire adjacency matrix, this method introduces a random sampling strategy for feature maps to reduce redundant computation and improve search efficiency. In the validation phase, i.e. Figure 2The blue section illustrates how the gait emotion recognition method based on neural network architecture search stacks optimized modules obtained from the search to construct the final network structure. This model serves as the final method for solving emotion recognition tasks based on skeletons. The model was trained from scratch to validate the automatically constructed model structure of the proposed gait emotion recognition method based on neural network architecture search, by extracting skeleton gait sequences. The emotional representations contained within the skeleton are used to address the effectiveness of skeleton-based emotion recognition tasks.

[0118] Constructing a reasonable search space is crucial for discovering the optimal module structure and stacking it to build the final network architecture. Since the gait emotion recognition method based on neural architecture search is the first neural architecture search (NAS) framework proposed for gait-based emotion recognition, no existing method can directly provide a search space highly matched to this task. Therefore, this invention designs a novel search space encompassing seven operations: Temporal Convolutional Network (TCN), four Spatio-Graph Convolutional Network (Spatio-GCN) operations, zero operation, and skip-connection. Optimized operations are selected for extracting emotional features between different units. Simultaneously, the connections between operations are preserved to improve the flexibility and expressiveness of the space. In the definition of the module structure, the first and second input nodes (internal nodes) of each cell correspond to the outputs of the previous two cells, respectively. The output node of the cell is formed by combining the feature maps of all its intermediate nodes through depthwise concatenation, as follows: Figure 2 The transformation process of pink rectangles 1 and 2 is shown in the diagram. Based on the search space designed above, the gait emotion recognition method based on neural network architecture search employs a gradient-based optimization strategy to automatically search for the optimal module structure and then constructs the final network architecture through module stacking, thereby achieving effective recognition of human emotional states.

[0119] This invention introduces a neural network architecture search method into gait emotion recognition for the first time, successfully applying an automated model building approach in this field. This automated modeling framework significantly improves model building efficiency for specific data types and application scenarios, while also enabling customized structural design based on the feature differences inherent in different data, thereby enhancing the model's expressive power and relevance. Existing gait emotion recognition methods are systematically reviewed, and a broad-coverage module search space is constructed accordingly, providing a rich and detailed set of candidate structures for gait emotion recognition methods based on neural network architecture search during the search phase. This search space design effectively ensures that the model can generate the optimal gait emotion representation extraction module during the automated construction process, thereby improving the overall recognition accuracy in emotion classification tasks. To improve the efficiency of the neural architecture search phase, a candidate operation evaluation strategy based on a noise smoothing mechanism is proposed. This strategy, by introducing regularization during the search process, effectively alleviates the unfairness between candidate operations, enhancing the stability and generalization ability of the search results. Furthermore, this method significantly reduces computational resource consumption and accelerates the convergence speed of the entire search process while maintaining performance.

[0120] To verify the effectiveness and advancement of the proposed method, extensive experiments were conducted on widely used gait emotion datasets for the proposed gait emotion recognition method based on neural network architecture search, and comparisons were made with current state-of-the-art gait emotion recognition methods. Classification accuracy was used for evaluation. Experimental results show that the proposed method can achieve optimal emotion category classification accuracy by automatically designing a gait emotion classification model. Furthermore, the gait emotion recognition method based on neural network architecture search consumes only 0.6 GPUs / hour for automatic model building, significantly improving the efficiency of designing model structures from scratch, and playing a crucial role in advancing the field of gait emotion recognition.

[0121] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0122] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A gait emotion recognition method based on neural network architecture search, characterized in that, The method comprises the following steps: acquiring a three-dimensional skeleton gait sequence, and representing each frame of the gait skeleton sequence as a gait skeleton graph; a search space comprising seven candidate operations is constructed, a directed acyclic graph is constructed by a sequential greedy search strategy, and a module structure is generated; a TCN noise disturbance is used to optimize the module structure, a skip connection operation is introduced, and the search capability is optimized by random sampling of feature channels; the seven candidate operations comprise a time series convolution operation, four spatial graph convolution operations, no operation, and a skip operation; the directed acyclic graph comprises intermediate nodes for storing the space-time features of the gait graph sequence, edges for determining the information flow direction from a previous intermediate node to a next intermediate node, edge operations for processing the data transmitted by the edges, and operation weights for controlling the adaptability of different operations to the skeleton features; the sequential greedy strategy performs a phased optimization mechanism, iteratively optimizes the intermediate node connection, the edge operation, and the operation weight in sequence, and makes a pruning decision based on three indexes of candidate operation importance, selection certainty, and stability; the specific implementation process of making a pruning decision based on the three indexes of candidate operation importance, selection certainty, and stability comprises: the importance of the candidate operation on the edge is calculated according to the architecture parameters of the candidate operation on the edge, and the specific formula is as follows: where o denotes candidate operations selected from all candidate operation sets not containing zero operation, a o denotes an architecture parameter on the edge for measuring operation weight of candidate operation o, O denotes a set of all candidate operations on the edge, zero denotes a zero operation, f denotes a softmax function, o` denotes candidate operations selected from all candidate operation sets containing zero operation; the zero operation represents that the connection does not exist; The normalized weight p of the non-zero operation o As a distribution, the calculation formula is as follows: The complement of the normalized entropy of the operation distribution is obtained by combining the entropy calculation operation, and a candidate operation selection certainty S is obtained for measuring the concentration degree of the non-zero operation on the side SOC The calculation formula is as follows: wherein, log(|O|-1) represents the maximum possible entropy when all non-zero operations are uniformly distributed; The average stability selected by the edge operation is calculated using a histogram intersection, obtaining the selected candidate operation stability S SOS , as follows: where t represents the iteration number, T represents the total iteration number in the search process, K represents any iteration number in [0, T], represents the normalized weight of the selected operation o when the iteration number is t, represents the normalized weight of the selected operation o when the iteration number is T; According to the above three indexes, the score S of the candidate operation is calculated score The calculation formula is as follows: S score = Norm(S COI ) x Norm(S SOC ) x Norm(S SOS ) wherein, Norm() represents a maximum-minimum value normalization operation; the candidate operations are sorted in descending order according to the scores of the candidate operations, and the first candidate operation is selected as a deterministic operation; when two deterministic operations have been selected for an intermediate node, the candidate operations represented by other incoming edges will be pruned to generate a module structure; based on the optimized module structure, a graph convolution network for gait emotion recognition is constructed, a gait skeleton graph is input, and an emotion classification result is output.

2. The gait emotion recognition method based on neural network architecture search according to claim 1, characterized in that, the specific implementation process of optimizing the module structure by using a TCN noise disturbance comprises: a TCN noise disturbance is used to intervene in the TCN operation to obtain the selection certainty of the optimized candidate operation, and the intervention formula is as follows: where S COI 1 represents the selection certainty of the candidate operation after adding the TCN disturbance, represents a sigmoid function adding θ as a weight, θ represents a function of regularizing architecture parameter weight, F(α o ) represents an influence function on the architecture parameter α o , represents the output of the optimized f function, λ represents a proportion used to balance the architecture parameter and the validation loss for determining the final operation, η α represents a learning rate of the architecture parameter α, p o ’ represents the normalized weight of the non-zero operation after adding the TCN disturbance, f(α o＝tcn ) represents the output after performing a softmax operation on the architecture parameter α o＝tcn , and x represents the input sentiment feature, represents Gaussian noise with zero mean and small variance; the selection certainty and the stability of the optimized candidate operation are calculated, a pruning decision is made in combination with the selection certainty of the optimized candidate operation, and an optimized module structure is obtained.

3. The gait emotion recognition method based on neural network architecture search according to claim 2, characterized in that, the specific implementation process of introducing a skip connection operation and optimizing the search capability by random sampling of feature channels comprises: an auxiliary skip connection operation is introduced, the skip connection operation comprises a TGCN and an identity mapping, the candidate search space is modified, and the modification formula is as follows: wherein, denotes the modified candidate search space, and γ is a weight parameter for the auxiliary connection. the output of each node partial feature channel is calculated by a sampling mechanism: wherein, M represents the number of candidate operations, L represents the probability of a selected channel, Lx x and (1-L) x x represent the sampled channel and the non-sampled channel respectively, and k represents the kth candidate operation that is not a TCN.

4. The gait emotion recognition method based on neural network architecture search according to claim 1, characterized in that, further comprising selecting the optimized candidate operation for extraction of emotion features between different units, the first and second input nodes of each unit correspond to the outputs of the previous two units respectively, and the output node of the unit is composed of the feature maps of all intermediate nodes of the unit by deep concatenation.

Images