Neural network-based open-set behavior intention recognition method and system, and electronic device

Abstract

The application discloses a behavior intention open set recognition method and system based on a neural network and electronic equipment, and belongs to the field of intention recognition. The method firstly adopts a known class classification network comprising a self-attention hybrid convolution module and a time sequence change perception module to respectively extract key local features and global time sequence changes in the time dimension, and process the same into low-dimensional class activation vectors; then, in the training process, an adaptive threshold detection module is used to fit Weibull distribution of each action intention class and an unknown action intention class judgment threshold, and in the test or reasoning stage, the class activation vectors of test samples are corrected, and samples less than the threshold of the corresponding class are identified as the unknown action intention class. Action intention open set recognition experiments on various time sequence classification data sets show that the overall performance of the method is better than that of each baseline method, and ablation experiments of the model prove the rationality of the network structure.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intention recognition, and in particular to a behavior intention open set recognition method and system based on a neural network and an electronic device. BACKGROUND

[0002] Action intention refers to the internal will or purpose expressed by an individual when performing an action, and is an important part of action recognition. Common action intention recognition scenarios include speech recognition, gesture recognition, and body movement recognition. In the field of action recognition, traditional recognition methods mainly rely on related features and rules defined by experts. However, actions are often complex and variable, making it difficult to define related features and rules for all action types. Therefore, traditional methods often perform poorly in complex and variable scenarios.

[0003] In recent years, with the development of neural network technology represented by deep learning, action intention recognition technology based on neural networks has made significant progress. However, when training a neural network model to handle action intention recognition, it often relies on a large amount of manually labeled data of action time series. However, action types are often difficult to completely enumerate, so if some actions have not been labeled and effectively trained, there may be a problem of identifying unknown action intentions as known intentions.

[0004] Time series open set recognition is an extension of time series classification tasks and has become one of the research hotspots in time series analysis in recent years. When performing time series classification, in addition to known class samples, there may also be unknown or abnormal samples. This open scenario often occurs in reality, for example, due to device changes or failures, resulting in unknown time series samples. Traditional time series classification algorithms can only classify known classes and cannot identify unknown or abnormal samples, which affects the accuracy and reliability of the classification model.

[0005] Open set recognition technology can be considered an effective solution to identifying unknown actions in the process of action intention recognition. However, in the current prior art, there is still no good solution to how to introduce open set recognition technology to identify unknown actions. SUMMARY

[0006] To overcome the above-mentioned deficiencies in the prior art that it is difficult to accurately identify new unknown intention actions in action sequences, the present application provides a behavior intention open set recognition method and system based on a neural network and an electronic device, which can effectively solve the above-mentioned problems.

[0007] The technical solutions specifically adopted by the present application are as follows:

[0008] In a first aspect, the present application provides a behavior intention open set recognition method based on a neural network, which includes the following steps:

[0009] S1, for the time series sample set recording human behavior and its artificial annotation results, add the time series sample with marked intention type to the known class classification data set, add the time series sample failing to mark the intention type to the unknown class set, thereby constructing the open set recognition data set;

[0010] S2, using the known class classification data set obtained in S1, training the known class classification model until the model converges, so that it can accurately classify the input known class time series sample; in the known class classification model, the time series input into the model is extracted key local features through two cascaded self-attention hybrid convolution modules, and global time sequence features are extracted through a time sequence change perception module; after the key local features and the global time sequence features are fused, the corresponding classification label is output through the classification layer;

[0011] S3, after the known class classification model in S2 is trained, the self-adaptive threshold open set recognition module processes all time series samples in the known class classification data set that can be correctly classified by the known class classification model into class activation vectors, and calculates the mean center of the class activation vector of each known class; for each intention class, select part of the activation vector farthest from the mean center of the class activation vector, and fit its Weibull distribution; for each time series sample in the known class classification data set, calculate the Weibull probability correction weight vector according to the Weibull distribution of the intention class to which it belongs, correct the class activation vector and normalize it through the Softmax function to obtain the class probability correction vector; using the open set recognition data set obtained in S1, searching for the best unknown class judgment threshold of each known intention class according to the accuracy of open set recognition, and then forming an unknown class judgment threshold vector;

[0012] S4, the time series of recording human behavior to be recognized intention type is processed into a class probability correction vector in the same way as the time series sample in S3, and then the intention class with the maximum probability value in the class probability correction vector is determined; if the probability value corresponding to the intention class is greater than the best unknown class judgment threshold corresponding to the intention class in the unknown class judgment threshold vector, the intention class is taken as the intention class of the time series to be recognized, otherwise the time series to be recognized is recognized as an unknown class.

[0013] As a preferred embodiment of the first aspect, the model input of the known class classification model is a time series recording human behavior, the input of the first self-attention hybrid convolution module is the time series, the input of the second self-attention hybrid convolution module is the output of the first self-attention hybrid convolution module, the output of the second self-attention hybrid convolution module is stretched into a one-dimensional vector to obtain a first output signal; the time series of the model input is input into the bidirectional LSTM network through the time series change perception module, the global attention mechanism is applied to the signals of each hidden step of the bidirectional LSTM network to extract a time series context vector, then the time series context vector is spliced with the last signal of the hidden layer of the bidirectional LSTM network to obtain a second output signal; finally, the first output signal is input into the first full connection layer, the second output signal is input into the second full connection layer, and the two are spliced and input into the third full connection layer to obtain a class activation vector, and the class activation vector is input into the Softmax function to obtain the intent category output by the known class classification model.

[0014] As a preferred embodiment of the first aspect, the first self-attention hybrid convolution module and the second self-attention hybrid convolution module adopt the same module structure, including a hybrid one-dimensional convolution layer, a self-attention layer, a residual connection layer and a one-dimensional average pooling layer.

[0015] The hybrid one-dimensional convolution layer first performs one-dimensional convolution on the module input X using four one-dimensional convolution kernels with different widths, and splices the convolution results of the four widths to obtain a spliced result.

[0016] The self-attention layer performs self-attention weighting on the spliced result to obtain an output signal.

[0017] The residual connection layer superimposes the module input and the output signal after each one-dimensional convolution layer to obtain a residual connection result.

[0018] The one-dimensional average pooling layer performs average pooling on the residual connection result to obtain a module output.

[0019] As a preferred embodiment of the first aspect, in the time series change perception module, the time series of the model input is input into the bidirectional LSTM network layer, the positive and negative hidden layer signals of each time step of the bidirectional LSTM network layer are spliced and input into the global attention mechanism layer, the spliced hidden layer signal is weighted and activated by the Tanh function, and the weight value is calculated by the Softmax function, and finally the spliced hidden layer signals of all time steps are weighted and summed to obtain a time series context vector; the time series context vector is spliced with the spliced hidden layer signal of the last time step of the hidden layer of the bidirectional LSTM network to obtain a global time series feature as the module output.

[0020] As a preferred embodiment of the first aspect, the loss function used in training the known class classification model is cross-entropy loss, and the AdamW optimizer is used to update all trainable parameters of the model.

[0021] As a preferred embodiment of the first aspect, in the S4, when calculating the category probability correction vector, the time series of the record human behavior of the to-be-identified intention type is input into the trained known class classification model, a category activation vector is output by the third fully connected layer, then the Weibull probability correction weight vector is calculated according to the category Weibull distribution obtained in the S3, and the category activation vector is further weighted and corrected by using the calculated Weibull probability correction weight vector, and the correction result is normalized by using the Softmax function to obtain the category probability correction vector.

[0022] As a preferred embodiment of the first aspect, the number of convolution kernels of the known class classification network is 16.

[0023] As a preferred embodiment of the first aspect, the number of hidden layer units of the bidirectional LSTM network layer of the known class classification network, the number of hidden layer units N1 of the first fully connected layer and the second fully connected layer are 32.

[0024] As a preferred embodiment of the first aspect, the number of Weibull distribution fittings of the adaptive open set recognition module is 20.

[0025] As a preferred embodiment of the first aspect, the search step length s of the hyperparameter b of the adaptive open set recognition module is 0.25.

[0026] As a preferred embodiment of the first aspect, when the multi-element time series prediction model is trained, the AdamW optimizer is used to update all trainable parameters of the neural network model, the learning rate a is 0.001, and the decay weight b is 0.0001.

[0027] In a second aspect, the present application provides a time series open set recognition system based on a neural network, which comprises:

[0028] A recording module is configured to obtain a time series of record human behavior of a to-be-identified intention type.

[0029] A recognition module is configured to determine whether the to-be-identified time series belongs to a known intention category or an unknown category according to the behavior intention open set recognition method based on a neural network according to any one of the embodiments of the first aspect.

[0030] In a third aspect, the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions are executed by a processor to implement the behavior intention open set recognition method based on a neural network according to any one of the embodiments of the first aspect.

[0031] In a fourth aspect, the present application provides a computer readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the neural network-based open-set recognition method for behavior intention according to any one of the first aspect is implemented.

[0032] In a fifth aspect, the present application provides a computer electronic device, characterized in that comprising a memory and a processor.

[0033] The memory is configured to store a computer program.

[0034] The processor is configured to implement the neural network-based open-set recognition method for behavior intention according to any one of the first aspect when executing the computer program.

[0035] Compared with the prior art, the neural network-based open-set recognition method for behavior intention has the following beneficial effects:

[0036] The neural network-based open-set recognition method for behavior intention comprises a known class classification and an open-set recognition, wherein the former is configured to extract key local features and global time sequence features of the action time sequence, and process them into low-dimensional class activation vectors, and the latter is a self-adaptive threshold open-set recognition module configured to fit a Weibull distribution of each known action intention class and automatically search for a corresponding unknown class judgment threshold, correct the class activation vector of the sample in the open-set test, and identify the sample smaller than the corresponding threshold as an unknown action intention class, so that the method can effectively identify unknown classes while ensuring the accuracy of known classes. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a step flowchart of the neural network-based open-set recognition method for behavior intention.

[0038] Figure 2 It is a model structure schematic diagram of the open-set recognition method for behavior intention.

[0039] Figure 3 It is a structure schematic diagram of the open-set recognition model for behavior intention composed of the known class classification model and the self-adaptive threshold open-set recognition module.

[0040] Figure 4 It is a module composition schematic diagram of the neural network-based open-set recognition system for behavior intention.

[0041] Figure 5 It is an open-set recognition confusion matrix result in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many different ways other than those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, so the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0043] As shown in the preferred embodiment of the present application, the above-mentioned neural network-based behavior intention open set recognition method specifically includes steps S1-S4, and the specific implementation of each step is described below. Figure 1

[0044] S1, for the time series sample set recording human behavior and its artificial annotation results, the time series sample with marked intention type is added to the known class classification data set, and the time series sample that fails to mark the intention type is added to the unknown class set, thereby constructing an open set recognition data set.

[0045] It should be noted that the specific data set type in the present application needs to be determined according to the action intention recognition scene. For example, in the subsequent embodiments, the recognition scene can be speech recognition, gesture recognition, body action recognition, etc. For the speech recognition scene, the time series sample recording human behavior is a video of a person reading Arabic numerals in Arabic, and the corresponding intention label is the Arabic numerals expressed by the person; for the gesture recognition scene, the time series sample recording human behavior is a video of a person making gesture movements, and the corresponding intention label is the gesture category expressed by the person; for the body action recognition scene, the time series sample recording human behavior is the gyroscope time series data when the person is walking, sitting, standing, lying down, etc., and the corresponding intention label is the action currently being performed by the person.

[0046] In the embodiments of the present application, the construction method of the above-mentioned open set recognition data set specifically includes the following sub-steps:

[0047] S11, for the target action intention recognition scene, manually screening out a time series sample set X k ={X1,X2,…,X D}∈R {D×T} , which contains K known categories, T is the length of the time series sample, D is the number of known category samples, and the corresponding action intention label is Y k ={y1,y2,…,y D}∈R {D} y i ​∈[0,K-1], given the class classification dataset D k ={X k ;Y k};

[0048] S12, The remaining time-series sample set of special or unknown action intentions X uk ={X1,X2,…,X S}∈R {S ×T} T is the length of the time series sample, S is the number of known category samples, and the corresponding action intent label is Y. uk ={y1,y2,…,y S}∈R {S} ,y i =K, the set of unknown classes is D uk ={X uk ;Y uk};

[0049] S13. Combine the known class classification dataset D obtained from S11 and S12. k and the unknown class set D uk By mixing, we obtain the open set recognition dataset D. o ={D k ;D uk The class labels in the dataset are one-hot encoded to facilitate model training.

[0050] S2. Using the known class classification dataset obtained in S1, train the known class classification model until the model converges, enabling it to accurately classify the input known class time series samples. In the above known class classification model, the input time series passes through two cascaded self-attention hybrid convolutional modules to extract key local features, and through a time-series change awareness module to extract global temporal features. The key local features and global temporal features are fused and then passed through the classification layer to output the corresponding classification label.

[0051] The model input of the known class classification model is a time sequence recording human behavior (training is for a time sequence sample, and the inference stage is a time sequence to be identified), the input of the first self-attention hybrid convolution module is a time sequence, the input of the second self-attention hybrid convolution module is the output of the first self-attention hybrid convolution module, the output of the second self-attention hybrid convolution module is stretched into a one-dimensional vector to obtain a first output signal. In addition, the time sequence change perception module inputs the time sequence of the model input (training is for a time sequence sample, and the inference stage is a time sequence to be identified) into a bidirectional LSTM network, applies a global attention mechanism to the signals of each hidden step of the bidirectional LSTM network to extract a time sequence context vector, then splices the time sequence context vector and the last signal of the hidden layer of the bidirectional LSTM network to obtain a second output signal; finally, the first output signal passes through a first full connection layer, the second output signal passes through a second full connection layer, and after splicing, the two pass through a third full connection layer to obtain a class activation vector, and after the class activation vector passes through a Softmax function, an intent category output by the known class classification model is obtained.

[0052] In addition, in the embodiments of the present application, the first self-attention hybrid convolution module and the second self-attention hybrid convolution module adopt the same module structure, specifically as shown in Figure 2 The mixed one-dimensional convolution layer first uses four one-dimensional convolution kernels with different widths to perform one-dimensional convolution on the module input, splices the convolution results of the four widths, and obtains a splicing result; the self-attention layer performs self-attention weighting on the splicing result to obtain an output signal; the residual connection layer superimposes the module input and the output signal after each one-dimensional convolution layer to obtain a residual connection result; and the one-dimensional average pooling layer performs average pooling on the residual connection result to obtain the module output. The two self-attention hybrid convolution modules are cascaded, the output of the previous module is used as the input of the next module, and the output of the second module is the extracted key local feature.

[0053] In addition, in the embodiments of the present application, in the time sequence change perception module, the time sequence of the model input is input into a bidirectional LSTM network layer, the concatenated hidden layer signals of each time step of the bidirectional LSTM network layer pass through a global attention mechanism layer, the weighted and Tanh activated output of the spliced hidden layer signals is calculated by a Softmax function to obtain a weight, and finally all the time steps of the spliced hidden layer signals are weighted and summed to obtain a time sequence context vector; the time sequence context vector is spliced with the spliced hidden layer signal of the last time step of the hidden layer of the bidirectional LSTM network to obtain a global time sequence feature as the module output.

[0054] The known class classification model needs to be trained in advance. In the embodiment of the present application, the loss function used in the training is cross-entropy loss, and the AdamW optimizer is used to update all trainable parameters of the model. In order to better understand the specific implementation process of S2 step, the specific processing process and training process in the known class classification model are described in detail as follows, specifically as shown in S21-S27.

[0055] S21, input the time series sample X of the record action of the input model into the first self-attention hybrid convolution module to extract key local features, which specifically includes the following S211-S214 sub-steps:

[0056] S211, use four one-dimensional convolution kernels with widths of 3, 5, 7 and 11 respectively in the mixed one-dimensional convolution layer to perform one-dimensional convolution on the time series sample X, and the number of each convolution kernel is u, and the convolution process is represented as:

[0057]

[0058] where * represents a one-dimensional convolution operation with equal-width padding, represents the trainable parameters of the convolution kernel in the mixed one-dimensional convolution layer, k is the width of the convolution kernel, and concat(·) represents splicing the convolution results of the four width convolution kernels;

[0059] S212, the self-attention layer obtains the output signal Z M from S211 to calculate the query matrix E, the key matrix K and the value matrix V:

[0060] E=W E Z M , K=W K Z M , V=W V Z M

[0061] where W E , W K and W V are the trainable parameter matrices of the layer, each matrix has a size of 4u×u, and the self-attention score matrix A is calculated by the query matrix E and the key matrix K, and the value matrix V is calculated to obtain the output of the final self-attention layer:

[0062]

[0063] Z A =AV

[0064] After the above calculation, the output signal of the self-attention layer is Z A ∈R {T×u} ;

[0065] S213, the residual connection layer obtains the output signal Z A and the time series sample input X are respectively subjected to one-dimensional convolution layers and then superimposed:

[0066]

[0067] wherein the signal Z A The convolution operation uses u one-dimensional convolution kernels with a width of 3, and its trainable parameters are u one-dimensional convolution kernels with a width of 1, and its trainable parameters are u one-dimensional convolution kernels with a width of 1;

[0068] S214, the output signal Z C is finally output through a one-dimensional average pooling layer, which is used to compress the information in the time dimension, and the output signal

[0069]

[0070] The output after the average pooling layer is reduced to half of the original input X in the time dimension, and the length of the signal in the time dimension will continue to decrease with the stacking of the modules. Adding the pooling layer can effectively reduce the number of parameters of the subsequent network layer, which not only reduces the calculation amount, but also effectively prevents the overfitting phenomenon of the model.

[0071] S22, the output signal is input into the second self-attention hybrid convolution module to extract key local features. The specific sub-steps can also refer to S211-S214, and the structures of the two modules are consistent. The output signal of the former module will be used as the input X of the latter module. After stretching, the output signal

[0072] S23, the time series sample X input into the model is input into the time series change perception module to extract global time series features, which includes the following S231-S233 sub-steps:

[0073] S231, the time series sample X is input into a bidirectional LSTM network layer with N1 hidden layer units:

[0074]

[0075]

[0076] wherein represents the hidden layer signal of the forward time step and the hidden layer signal of the reverse time step The output of the bidirectional LSTM is represented as

[0077] S232, the time series context vector C obtained in S231 is concatenated with the hidden layer signal of the last time step of the bidirectional LSTM network obtained in S231 The global attention mechanism is applied, and the weighted hidden layer signal is obtained as The weight a is calculated by the Softmax function i The time series context vector C is obtained by weighting and summing the hidden layer signals and the weight a i :

[0078]

[0079]

[0080]

[0081] S233, the time series context vector C obtained in S232 is concatenated with the hidden layer signal of the last time step of the bidirectional LSTM network obtained in S231 The output after concatenation is:

[0082]

[0083] S24, the output signal H obtained in S22 and the output H obtained in S233 T are input into the first fully connected layer and the second fully connected layer respectively, and the output signal H is obtained after concatenating the two: D

[0084]

[0085] where W C and b C are the trainable parameters and bias terms of the first fully connected layer, and W T and b T are the trainable parameters and bias terms of the second fully connected layer, and the number of hidden layer units of the first fully connected layer and the second fully connected layer is N1;

[0086] S24, the output signal H obtained in S24 D is input into the third fully connected layer, and the class activation vector q of the time series sample X is obtained:

[0087] q = W D · H D + b D

[0088] where W​D and b D are trainable parameters and bias terms of the third fully connected layer, and the number of hidden layer units is the number K of known classes;

[0089] S25, the class activation vector q of the time series sample X obtained in S25 is subjected to a Softmax function to obtain the classification probability p of the time series sample X:

[0090] p = softmax(q)

[0091] S26, the class label of the input sample is one-hot encoded, and the classification loss is calculated using a cross-entropy loss function with the classification probability p obtained in S25:

[0092]

[0093] where y c represents the encoding value of the corresponding class after one-hot encoding of the sample label.

[0094] S27, the random gradient of the model is calculated according to the neural network prediction loss L obtained in S26, and the AdamW optimizer is used to update all trainable parameters of the neural network model with a learning rate a and a decay weight b. The training process is iterated until the model converges.

[0095] S3, after the known class classification model in S2 is trained, the adaptive threshold open set recognition module processes all time series samples in the known class classification data set that can be correctly classified by the known class classification model into class activation vectors, and calculates the mean center of the class activation vectors of each known class; for each intent class, select the part of the activation vector farthest from the mean center of the class activation vector, and fit its Weibull distribution; for each time series sample in the known class classification data set, calculate the Weibull probability correction weight vector according to the Weibull distribution of the intent class to which it belongs, correct the class activation vector, and normalize it through the Softmax function to obtain the class probability correction vector; using the open set recognition data set obtained in S1, search for the best unknown class judgment threshold of each known intent class according to the accuracy of open set recognition, and then form an unknown class judgment threshold vector.

[0096] In the embodiments of the present application, the specific processing process in the adaptive threshold open set recognition module of the above S3 step is as follows:

[0097] S31, the known class classification data set D kAll time series samples that can be correctly classified are processed into category activation vectors (category activation vectors q can be output by the third fully connected layer in the known class classification model), and the category activation vector mean center μ of all category activation vectors in each known action intention category is calculated i , i ∈ [0, K-1];

[0098] S32, the category activation vector mean center μ of each intention category obtained according to S31 i , i ∈ [0, K-1], the activation vectors of η time series samples farthest from the category activation vector mean center in each category are selected for fitting the Weibull distribution of each category:

[0099]

[0100] Where k i , λ i are the shape and scale parameters of the Weibull distribution ρ i of the i-th intention category, τ i is the data transformation vector fitted, is the average vector of the category activation vectors of the η time series samples farthest from μ i in the current category, FitHigh(·) is the LibMR function for calculating the Weibull distribution, η represents the number of Weibull distribution fittings, dist(·) represents the calculation of and μ i , and the spatial distance is calculated as follows:

[0101]

[0102] Where represents the calculation of and μ i , and the standard Euclidean distance is calculated, represents the calculation of and μ i , and the cosine similarity is calculated;

[0103] S33, according to the Weibull distribution ρ i (τ i , k i , λ i ) of each category obtained in S32, i ∈ [0, K-1], the Weibull probability correction weight is calculated:

[0104]

[0105] In the formula: s(i) represents the calculation of the above mean center μ iand the distance indexes after the distances are sorted in descending order; K represents the number of action intention categories; k θ(i) , λ θ(i) and τ θ(i) respectively represent the shape parameter, the scale parameter and the fitted data transformation vector of the Weibull distribution corresponding to the distance index s(i); q represents the correct category activation vector output by the third fully connected layer in S24.

[0106] S34, the Weibull probability correction weight obtained in S33 is used to correct the category activation vector of the time series sample, and the Softmax function is used for normalization:

[0107]

[0108]

[0109] wherein, represents the corrected category activation vector of the i-th action intention category; q i represents the category activation vector of the i-th action intention category; w i represents the Weibull probability correction weight of the i-th action intention category.

[0110] S35, the best parameter b is selected according to the open set recognition accuracy of the module on the training set, that is, the b value is adjusted so that the accuracy of the open set recognition dataset D o obtained in S13 reaches the maximum, so as to determine the unknown category judgment threshold vector ∈ = [∈0, ∈1, …, ∈ K-1 ]. In the present application, any element ∈ i in the vector is determined by the following formula:

[0111]

[0112] wherein m i represents the number of all correctly classified samples of category i in the training set, is the probability value of all correctly classified samples of the action intention category i in the training set; mean() and std() respectively represent the mean function and the variance function; the search range of b is [0, 5], and the search step is an optimizable parameter s. According to the b value in the search range, the recognition accuracy of the open set recognition dataset D o reaches the maximum, the best unknown category judgment threshold ∈ i of the i-th action intention category in the above formula is calculated, and then the unknown category judgment threshold vector ∈ is composed. In this way, the module will automatically adapt to the activation vector probability threshold distribution of each known category, and the grid search of the OpenMax algorithm for parameters is simplified.

[0113] In this embodiment, a schematic diagram of the behavior intent open set recognition model structure, consisting of a known class classification model and an adaptive threshold open set recognition module, is shown below. Figure 3 As shown. Based on the threshold vector determined by the unknown category, the method for identifying known and unknown classes can be expressed by the following action intent category recognition formula:

[0114]

[0115] in This is the class probability correction vector for the sample. represent The intent category with the highest probability value. represent The highest probability value in, The identified action intent category label, where This indicates that the action intent belongs to the unknown action intent category. This action intent category recognition formula can be applied to the actual reasoning process, namely step S4.

[0116] S4. Process the time series of human behavior records of the intent type to be identified into a category probability correction vector in the same way as the time series samples in S3. Then, determine the intent category with the highest probability value according to the category probability correction vector. If the probability value corresponding to the intent category is greater than the best unknown category judgment threshold corresponding to the intent category in the unknown category judgment threshold vector, then the intent category is taken as the intent category of the time series to be identified. Otherwise, the time series to be identified is identified as an unknown category.

[0117] In addition, in the embodiments of the present invention, when calculating the category probability correction vector for the time series of recorded human behavior of the intent type to be identified, it is similar to that of the sample. First, the time series of recorded human behavior of the intent type to be identified is input into the trained known class classification model, and the category activation vector is obtained by the output of the third fully connected layer. Then, according to the category Weibull distribution obtained in S3, the Weibull probability correction weight vector is calculated. Then, the calculated Weibull probability correction weight vector is used to weight and correct the category activation vector. The correction result is normalized by the Softmax function to obtain the category probability correction vector.

[0118] In embodiments of the present invention, during the testing or inference phase of open set identification, the specific processing procedure of S4 described above can be represented as follows:

[0119] S41. During the open set recognition test, based on the converged known class classification model obtained in S2, process the time series or time series test samples of the recorded human behavior of the intent type to be identified into class activation vectors q.

[0120] S42, according to the category Weibull distribution p obtained in S32 i (τ i ,k i ,λ i ), i ∈ [0, K-1] and the category activation vector q obtained in S41, calculate the Weibull probability correction weight vector w, correct the category activation vector q and normalize it through the Softmax function to obtain the category probability correction vector The specific process is as described in S33 and S34, which will not be repeated here.

[0121] S43, according to the category judgment threshold vector ∈ = [∈0, ∈1, …, ∈ K-1 ] obtained in S35 and the category probability correction vector The sample whose category probability correction vector is greater than or equal to its unknown category judgment threshold is identified as the known category, otherwise it is identified as the unknown category, as shown in the action intention category identification formula in S35.

[0122] It should be noted that the method steps of S1-S4 described above can be essentially realized in the form of a computer program.

[0123] Similarly, based on the same inventive concept, the present application also provides a neural network-based time series open set recognition system corresponding to the neural network-based time series open set recognition method provided in the above embodiment, as shown in Figure 4 , which comprises:

[0124] The recording module is configured to obtain a time series of recorded human behaviors of an intention type to be recognized.

[0125] The recognition module is configured to determine whether the time series to be recognized belongs to a known intention category or an unknown category according to the neural network-based behavior intention open set recognition method as described above.

[0126] Similarly, based on the same inventive concept, the present application also provides a computer electronic device corresponding to the neural network-based time series open set recognition method provided in the above embodiment, which comprises a memory and a processor.

[0127] The memory is configured to store a computer program.

[0128] The processor is configured to implement the neural network-based time series open set recognition method as described above when the computer program is executed.

[0129] In addition, the logic instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application.

[0130] Therefore, based on the same inventive concept, the present application provides a computer readable storage medium corresponding to the neural network based time series open set identification method, and the storage medium stores a computer program. When the computer program is executed by a processor, the neural network based time series open set identification method as described above can be realized.

[0131] Therefore, based on the same inventive concept, the present application provides a computer program product, including computer programs / instructions, which are executed by a processor to realize the neural network based time series open set identification method as described above.

[0132] Specifically, in the computer readable storage medium of the above three embodiments, the computer program stored therein is executed by a processor to perform the steps S1-S4.

[0133] It can be understood that the storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc. various media that can store program codes.

[0134] It can be understood that the processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0135] It should be further noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method described is only a logical functional division, and there can be another division mode in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0136] In order to show the advantages of the neural network-based behavior intention open set recognition method shown in S1-S4 of the above application, the following will apply it to a specific example to show its technical effect.

[0137] Embodiment

[0138] The steps of this embodiment are described in the foregoing S1-S4, which will not be described here. The following will show part of the implementation process and implementation results:

[0139] For the convenience of description, the method shown in the foregoing S1-S4 of the present application is referred to as AHCBL-OT. In this embodiment, the selection of the parameters of AHCBL-OT is as follows: the number of convolution kernels of the known class classification network is u=16, the number of hidden layer units of the bidirectional LSTM network layer of the known class classification network, the number of hidden layer units N1 of the first full connection layer and the second full connection layer is 32, the number of Weibull distribution fitting of the adaptive open set recognition module is η=20, and the search step length s of the hyperparameter b is 0.25. When the known class classification model is trained, the AdamW optimizer is used to update all trainable parameters of the neural network model, and the learning rate a is 0.001 and the decay weight β is 0.0001.

[0140] The experiment of the embodiment compares AHCBL-OT with several open set recognition methods, and the prediction methods used as controls are as follows: (1) W-SVM: including two SVM models, respectively learning the cumulative distribution function of the Weibull distribution of the rejection classification and the known class and the unknown class probability threshold. (2) EVM: based on extreme value theory, regarding the data set as a mixed distribution composed of known class samples and unknown class samples, and dividing the samples into known classes or unknown classes by selecting the smallest hyper-ellipsoid containing the known class samples. (3) CROSR: by constructing an auto-encoding network, obtaining a low-dimensional feature representation of the data, and inputting it into an unknown class detector based on OpenMax to realize open set recognition. (4) OS-InceptionTime: an unknown class detector designed based on a time series classification hybrid network (InceptionTime), which judges the unknown class according to the relationship between the input sample and the center of gravity of the known class and the DTW distance. (5) MEROS: based on a one-dimensional CNN, a deep auto-encoding network is constructed, and the maximum reconstruction error of the known class in the training set is used as the unknown class screening threshold.

[0141] In order to verify the technical effect of the technical scheme of the application, the macro-average F1 score (Macro-average F1-Score), the accuracy of the known class and the accuracy of the unknown class are used as the evaluation indexes of the method, and the calculation formula is as follows:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147] The embodiment selects four real action intention recognition scene time series classification data sets as open set recognition experiment data, which are: Spoken Arabic Digit (SAD) and UWave Gesture Library (UGL) from UEA, Mobile Health (MHealth) and Human Activity Recognition (HAR) from UCI, and the four data sets are as follows:

[0148] Spoken Arabic Digit (SAD): This dataset comes from the UEA dataset collection, which contains Mel-Frequency Cepstral Coefficients (MFCCs) time series corresponding to spoken Arabic digits, recorded by native Arabic speakers, with the number of classes corresponding to the Arabic digits.

[0149] UWave Gesture Library (UGL): This dataset comes from the UEA dataset collection, which contains a set of time series of gestures recorded by an accelerometer, with a total of 8 basic gestures, recording the position of the gesture in the X, Y, Z coordinates.

[0150] Mobile Health (MHealth): This dataset comes from the UCI repository, which contains the body motion and vital signs of 10 volunteers of different backgrounds during 12 physical activities, including time series of acceleration, turn rate, and magnetic field direction on the chest, right wrist, and left ankle.

[0151] Human Activity Recognition (HAR): This dataset comes from the UCI repository, which contains 3-axis linear acceleration and 3-axis angular velocity time series recorded by a smartphone worn on the waist of a volunteer during six activities (walking, walking upstairs, walking downstairs, sitting, standing, and lying).

[0152] In this embodiment, half of the classes in the above datasets are used as known classes, and the other half of the classes are used as unknown classes. The division of the training set and the test set is consistent with the original division of the dataset. All open set recognition models will be trained based on the training set containing only known classes, and tested using the test set containing both known and unknown classes during open set testing.

[0153] In this embodiment, the experimental results are listed in Table 1.

[0154] Table 1. Open set recognition results of all methods on 2 datasets

[0155]

[0156]

[0157] Overall, the four deep learning-based open-set recognition models (i.e., CROSR, OS-InceptionTime, MEROS, and AHCBL-OT) outperform the two machine learning models (i.e., W-SVM and EVM) in terms of macro-average F1 scores. W-SVM and EVM perform well on the SAD dataset but poorly on the remaining three datasets, indicating that they cannot well balance the accuracy of known classes and unknown classes. For example, on the UGL dataset, W-SVM has a high classification accuracy for known classes but a low recognition accuracy for unknown classes; on the contrary, EVM has a high recognition accuracy for unknown classes but a low accuracy for known classes, meaning that it misjudges some known class samples as unknown classes. These two machine learning models do not perform dimensionality reduction on time series data and are sensitive to parameter settings, which leads to this phenomenon.

[0158] In terms of model structure, CROSR and MEROS belong to reconstruction classification models based on auto-encoding networks, while OS-InceptionTime and the proposed AHCBL-OT belong to discriminative classification models. Compared with the other two deep learning-based baseline models, MEROS performs better than CROSR and OS-InceptionTime on the SAD, MHealth, and HAR datasets, and has a more balanced accuracy for known classes and unknown classes. The proposed AHCBL-OT model has a higher macro-average F1 value than other models on the first three datasets, indicating that AHCBL-OT can maintain a certain recognition ability for unknown classes while ensuring a high classification accuracy for known classes.

[0159] In addition, the proposed open-set recognition method is ablation studied on the SAD dataset to verify the effectiveness of the key structure of the model. In this embodiment, the adaptive threshold detection module in the AHCBL-OT method is replaced, i.e., the known class classification network is used to realize open-set recognition with the OpenMax open loss function, and this method is denoted as AHCBL-OM. Figure 5The open set recognition confusion matrix of two models is respectively shown, wherein "c0~c4" represents five known categories, and "uk" represents an unknown category. When the known category classification model is the same, the adaptive threshold detection module of the present application can recognize more unknown class samples than the OpenMax open set loss function, wherein the samples correctly classified into the unknown category reach 778. This shows that setting adaptive thresholds for different known categories balances the false positive rate and the false negative rate of the model to some extent. The ablation experiment above proves that the AHCBL-OT model designed in the present application can effectively obtain low-dimensional feature representations of time series, and the adaptive threshold detection module can better distinguish the class activation vectors from the classification network.

[0160] In summary, the present application proposes a neural network-based behavior intention open set recognition method. The known action intention category classification network of the method includes a self-attention hybrid convolution module and a time sequence change perception module, which respectively extract key local features and global time sequence changes in the time dimension and process them into low-dimensional class activation vectors. The adaptive threshold detection module fits the Weibull distribution of each action intention category and the unknown action intention category judgment threshold during the training process, and corrects the class activation vector of the test sample in the open set test stage, and identifies the sample less than the threshold of the corresponding category as the unknown action intention category. Action intention open set recognition experiments on various time sequence classification data sets show that the overall performance of the method is better than that of each baseline method, and the ablation experiment of the model proves the rationality of the network structure.

[0161] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the related art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A method for recognizing open sets of behavioral intentions based on neural networks, characterized in that, Includes the following steps: S1. For the time series sample set recording human behavior and its manual annotation results, add the time series samples with labeled intention types to the known class classification dataset, and add the time series samples without labeled intention types to the unknown class set, thereby constructing an open set recognition dataset. S2. Using the known class classification dataset obtained in S1, train the known class classification model until the model converges, enabling it to accurately classify the input known class time series samples. In the known class classification model, the input time series passes through two cascaded self-attention hybrid convolution modules to extract key local features, and through a time series change perception module to extract global time series features. The key local features and global time series features are fused and then passed through the classification layer to output the corresponding classification label. The known class classification model takes a time series of human behavior as its input. The first self-attention hybrid convolutional module takes the time series as its input, and the second self-attention hybrid convolutional module takes the output of the first self-attention hybrid convolutional module as its input. The output of the second self-attention hybrid convolutional module is stretched into a one-dimensional vector to obtain the first output signal. The temporal change perception module inputs the time series of the model input into a bidirectional LSTM network. It applies a global attention mechanism to the signals of each hidden step of the bidirectional LSTM network to extract the temporal context vector. Then, it concatenates the temporal context vector with the last signal of the hidden layer of the bidirectional LSTM network to obtain the second output signal. Finally, the first output signal is passed through a first fully connected layer, and the second output signal is passed through a second fully connected layer. The two are then concatenated and passed through a third fully connected layer to obtain the category activation vector. The category activation vector is then passed through a Softmax function to obtain the intent category output by the known class classification model. The first self-attention hybrid convolutional module and the second self-attention hybrid convolutional module adopt the same module structure, including a hybrid one-dimensional convolutional layer, a self-attention layer, a residual connection layer and a one-dimensional average pooling layer; The hybrid one-dimensional convolutional layer first uses four one-dimensional convolutional kernels of different widths to perform one-dimensional convolution on the module input, and then concatenates the convolution results of the four widths to obtain the concatenated result; The self-attention layer performs self-attention weighting on the splicing result to obtain the output signal; The residual connection layer superimposes the module input and output signals after they have passed through their respective one-dimensional convolutional layers to obtain the residual connection result. The one-dimensional average pooling layer performs average pooling on the residual connection results to obtain the module output; S3. After the known class classification model training in S2 is completed, the adaptive threshold open set recognition module processes all time series samples in the known class classification dataset that can be correctly classified by the known class classification model into class activation vectors, and calculates the mean center of the class activation vector for each known class. For each intent class, the activation vectors furthest from the mean center of its class activation vector are selected and fitted with its Weibull distribution. For each time series sample in the known class classification dataset, the Weibull probability correction weight vector is calculated according to the Weibull distribution of its intent class, and its class activation vector is corrected and normalized by the Softmax function to obtain the class probability correction vector. Using the open set recognition dataset obtained from S1, search for the optimal unknown category judgment threshold for each known intent category based on the accuracy of open set recognition; S4. Process the time series of human behavior records of the intent type to be identified into a category probability correction vector in the same way as the time series samples described in S3. Then, determine the intent category with the highest probability value according to the category probability correction vector. If the probability value corresponding to the intent category is greater than the optimal unknown category judgment threshold corresponding to the intent category, then the intent category is taken as the intent category of the time series to be identified; otherwise, the time series to be identified is identified as an unknown category.

2. The neural network-based open set recognition method for behavioral intent as described in claim 1, characterized in that, In the time-series change perception module, the time series input from the model is fed into a bidirectional LSTM network layer. The forward and reverse hidden layer signals of each time step of the bidirectional LSTM network layer are concatenated and then passed through a global attention mechanism layer. After the concatenated hidden layer signals are weighted and activated by Tanh, the weights are calculated by the Softmax function. Finally, the concatenated hidden layer signals of all time steps are weighted and summed to obtain the time-series context vector. The time-series context vector is concatenated with the concatenated hidden layer signal of the last time step of the bidirectional LSTM network hidden layer and then used as the global time-series feature output by the module.

3. The neural network-based open set recognition method for behavioral intent as described in claim 1, characterized in that, When training the known class classification model, the loss function used is cross-entropy loss, and the AdamW optimizer is used to update all trainable parameters of the model.

4. The behavior intent open set recognition method based on neural networks as described in claim 1, characterized in that, In step S4, when calculating the category probability correction vector, the time series of human behavior recorded for the intent type to be identified is first input into the trained known class classification model. The category activation vector is obtained by the output of the third fully connected layer. Then, based on the category Weibull distribution obtained in step S3, the Weibull probability correction weight vector is calculated. The calculated Weibull probability correction weight vector is then used to weight and correct the category activation vector. The correction result is normalized by the Softmax function to obtain the category probability correction vector.

5. A time series open set recognition system based on neural networks, characterized in that, include: The recording module is used to acquire time series of recorded human behaviors of the intent type to be identified; The identification module is used to determine whether the time series to be identified belongs to a known intent category or an unknown category according to the neural network-based open set identification method according to any one of claims 1 to 4.

6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the neural network-based open set recognition method for behavioral intent as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the neural network-based open set recognition method for behavioral intent as described in any one of claims 1 to 4.

8. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the neural network-based open set recognition method for behavioral intent as described in any one of claims 1 to 4 when executing the computer program.

Images