A small-sample behavior recognition and classification incremental learning method, device and readable medium

Abstract

The present invention discloses a small-sample behavior recognition and classification incremental learning method, device and readable medium, which constructs a first network model and a first classifier, a first feature extraction network and a first weight matrix generation network that are interconnected; obtains a first behavior data set and a second behavior data set, trains the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier; inputs the second behavior data set into the first feature extraction network, outputs the first weight matrix from the first weight matrix generation network, obtains a first incremental network model based on the first weight matrix, uses the second behavior data set to train the first feature extraction network, the first weight matrix generation network and the second classifier, uses the second behavior data set to train the second network model and the third classifier, and repeatedly trains the second incremental network model using the third behavior data set to obtain a target network model, thereby avoiding the problem of catastrophic forgetting.

Description

Technical Field

[0001] The present invention relates to the field of behavior recognition, and in particular to a method, device and readable medium for incremental learning of small sample behavior recognition classification. Background Art

[0002] Among human cognitive abilities, memory is one of the most relevant. However, memory is fragile. In particular, research in cognitive psychology has shown that old memories can be lost when new knowledge interferes with them, a process known as task interference. In the case of artificial neural networks, task interference is the root cause of the problem of catastrophic forgetting (CF). In practice, when a model learns a new task, its existing network parameters are extensively updated, leading to catastrophic forgetting of previously learned knowledge.

[0003] Behavior recognition tasks analyze collected behavioral action data and combine it with deep neural network algorithms to classify behavioral actions. However, since it's impossible to train all defined behaviors (such as walking up stairs, standing up, and drinking water) at once, only a limited number of behavioral categories can be recognized and classified.

[0004] Existing behavior recognition research methods mostly use time series models, regression models with spatiotemporal regularization, and external context feature models. In addition, models based on deep neural networks have significantly improved the performance of behavior recognition by more accurately representing nonlinear spatiotemporal correlations. However, the superior performance of these methods is based on large-scale training data, and they are unable to effectively recognize data of new behavior categories. Only by training existing network models with new behavior category data will cause the existing network models to forget a large amount of original knowledge, that is, catastrophic forgetting, making the trained network model unable to effectively classify the old behavior category data that has been learned. If all historical behavior data and new category behavior data are used together to train a new network model, the time cost is too high and requires a large amount of new category behavior data. Summary of the Invention

[0005] In response to the technical problems existing in the above background technology, the embodiment of the present application aims to propose a small sample behavior recognition and classification incremental learning method, device and readable medium to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, embodiments of the present application provide a small sample behavior recognition and classification incremental learning method, comprising the following steps:

[0007] S1, constructing a first network model and a first feature extraction network based on an LSTM structure, as well as a first classifier and a first weight matrix generation network based on a fully connected layer, wherein the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network;

[0008] S2, obtaining a first behavior data set and a second behavior data set with different categories, and training the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier;

[0009] S3, inputting the second behavior data set into the first feature extraction network, and obtaining the first weight matrix from the output of the first weight matrix generation network, forming a first incremental network model with the second network model processed by the first weight matrix, the first feature extraction network, and the first weight matrix generation network, and training the first feature extraction network, the first weight matrix generation network, and the second classifier of the first incremental network model using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network, and a third classifier;

[0010] S4: training the second network model and the third classifier using the second behavior data set to obtain a third network model and a fourth classifier;

[0011] S5, based on the first behavior data set and the second behavior data set, obtain a third behavior data set, repeat the process of step S3 for the third behavior data set to obtain a second incremental network model, and use the third behavior data set to train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute the target network model.

[0012] In some embodiments, the first behavior data set includes N1 categories of behavior data, with K1 pieces of behavior data in each category; the second behavior data set includes N2 categories of behavior data, with K2 pieces of behavior data in each category.

[0013] In some embodiments, step S5 obtains a third behavior data set based on the first behavior data set and the second behavior data set, specifically including: extracting (N1+N2) categories of behavior data from the first behavior data set and the second behavior data set as the third behavior data set, and each category of behavior data in the third behavior data set has K items, where K≤min{K1, K2}.

[0014] In some embodiments, step S3 inputs the second behavior data set into the first feature extraction network, and generates the first weight matrix from the first weight matrix output, specifically including:

[0015] Input the second line data set into the first feature extraction network to extract incremental data features;

[0016] The incremental data features are input into the first weight matrix generation network to obtain the first weight matrix.

[0017] In some embodiments, the second network model processed by the first weight matrix in step S3 is obtained by multiplying all parameters of the second network model by the first weight matrix.

[0018] In some embodiments, step S3 trains the first feature extraction network, the first weight matrix generation network, and the second classifier of the first incremental network model using the second behavior data set to obtain the second feature extraction network, the second weight matrix generation network, and the third classifier, specifically including:

[0019] The second behavior data set is input into the second network model of the first incremental network model after being processed by the first weight matrix, and the classification result is output through the second classifier; the first feature extraction network, the first weight matrix generation network and the second classifier are trained by gradient descent and back propagation to obtain the second feature extraction network, the second weight matrix generation network and the third classifier.

[0020] In some embodiments, the method further includes: using the third behavior data set as the first behavior training set and using the small sample behavior data of the newly added category as the second training set to repeat steps S3-S5 to obtain the final target network model.

[0021] In a second aspect, an embodiment of the present application provides a small sample behavior recognition and classification incremental learning device, comprising:

[0022] An initial model building module is configured to build a first network model and a first feature extraction network based on an LSTM structure, a first classifier based on a fully connected layer, and a first weight matrix generation network, wherein the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network;

[0023] A first training module is configured to obtain a first behavior data set and a second behavior data set with different categories, and train the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier;

[0024] a second training module configured to input the second behavior data set into the first feature extraction network, and obtain the first weight matrix from the output of the first weight matrix generation network, form a first incremental network model with the second network model processed by the first weight matrix, the first feature extraction network and the first weight matrix generation network, and train the first feature extraction network, the first weight matrix generation network and the second classifier of the first incremental network model using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network and a third classifier;

[0025] a third training module, configured to train the second network model and the third classifier using the second behavior data set to obtain a third network model and a fourth classifier;

[0026] The fourth training module is configured to obtain a third behavior data set based on the first behavior data set and the second behavior data set, repeat the process of the second training module on the third behavior data set to obtain a second incremental network model, and train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model through the third behavior data set to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute the target network model.

[0027] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The small sample behavior recognition and classification incremental learning method proposed in this paper can reuse the knowledge learned by the network model and avoid the catastrophic forgetting problem.

[0031] (2) The incremental learning algorithm adopted by the present invention prevents the catastrophic forgetting problem in the training process of new categories of behavioral data to a certain extent, and also avoids the requirement to use a large amount of new categories of behavioral data. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 is a diagram of an exemplary device architecture to which an embodiment of the present application may be applied;

[0034] Figure 2 Schematic diagram of the process of incremental learning method for small sample behavior recognition and classification according to an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of a network model of an incremental learning method for small sample behavior recognition and classification according to an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of an LSTM structure for an incremental learning method for small sample behavior recognition and classification according to an embodiment of the present invention;

[0037] Figure 5 Schematic diagram of a device for incremental learning of small sample behavior recognition and classification according to an embodiment of the present invention;

[0038] Figure 6 It is a structural diagram of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0040] Figure 1 An exemplary device architecture 100 is shown to which the small-sample behavior recognition and classification incremental learning method or small-sample behavior recognition and classification incremental learning device according to the embodiments of the present application can be applied.

[0041] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0042] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, 103.

[0043] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0044] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0045] It should be noted that the small sample behavior recognition and classification incremental learning method provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, and 103. Accordingly, the small sample behavior recognition and classification incremental learning device can be set in the server 105, or in the terminal devices 101, 102, and 103.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. If the processed data does not need to be acquired remotely, the above-described apparatus architecture may not include a network, but only require servers or terminal devices.

[0047] Figure 2 An embodiment of the present application provides a small sample behavior recognition and classification incremental learning method, including the following steps:

[0048] S1, construct a first network model and a first feature extraction network based on the LSTM structure, as well as a first classifier and a first weight matrix generation network based on a fully connected layer. The first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network.

[0049] In a specific embodiment, the specific network structure is as follows: Figure 3As shown, the first network model, the second network model, and the third network model mentioned in this application belong to Figure 3 The network model in the above is just a name under different training methods and states. The same principle applies to feature extraction networks, classifiers, weight matrix generation networks, etc. The behavior dataset has two input routes. On the one hand, the behavior dataset can be input into the first network model, and the output result of the first network model is then input into the first classifier to obtain the behavior classification result. On the other hand, the behavior dataset can be input into the first feature extraction network to extract incremental features, and the incremental features are input into the first weight matrix generation network to generate the weight matrix. The first network model F LSTM And the first feature extraction network F ex All use LSTM structure, such as Figure 4 As shown, where f t 、i t 、o t 、c t They represent the forget gate, input gate, output gate and cell state in LSTM respectively.

[0050] W f 、U f 、b f 、W t 、U t 、b t 、W c 、U c 、b c 、W o 、U o 、b o For the parameters that need to be trained in the following steps; the first weight matrix generates the network M FL and the first classifier C FL It is a fully connected layer, and the structure of each layer is y=Wx, that is, the lth layer is x l =W l-1 x l-1 , W is the parameter that needs to be trained in the following steps.

[0051] By adding these gate structures, LSTM can control the flow of information, that is, selectively store, lose, and update information at a certain moment, thereby solving the long-distance information transmission problem of RNN. However, it also leads to more parameters and longer training time.

[0052] Forget Gate f t Determines whether the information in the cell state is lost. This gate will input the input layer x t And the data t of the previous hidden layer t-1 , the output is a value in the interval [0,1] and is fed back to the cell state c t-1 . 1 means "completely retain" and 0 means "completely discard".

[0053] f t =σ(W f x t +U f h t-1 +b f );

[0054] Input gate i t Determine the data that needs to be updated in the cell state. Update the cell state and compare the previous state with f t Multiply to discard any information that needs to be discarded, and then add the newly updated cell state:

[0055] i t =σ(W i x t +U i h t-1 +b i );

[0056] Output gate o t Determine the output of some cell states, the calculation formula is as follows:

[0057] c t =f i c t-1 +i t ×(tanh(W c x t +U c h t-1 +b c ));

[0058] o t =σ(W o x t +U o h t-1 +b o );

[0059] h t =o t ×tanh(c t ).

[0060] S2, obtaining a first behavior data set and a second behavior data set of different categories, training the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier.

[0061] In a specific embodiment, the first behavior data set includes N1 categories of behavior data, each category of behavior data is K1 pieces, and the first behavior data set can be expressed as D old ={x i，j|1≤i≤N1,1≤j≤K1}. The second behavior dataset contains N2 categories of behavior data, each category has K2 pieces of behavior data, and the second behavior dataset can be expressed as D new ={x i，j |1≤i≤N2,1≤j≤K2}. The categories of the first and second rows of data sets are completely different. old The first network model is trained to obtain an initial network model after training, namely the second network model. At this time, the second network model has the ability to classify the behavior data included in the first behavior data set.

[0062] S3, input the second behavior data set into the first feature extraction network, and obtain the first weight matrix from the output of the first weight matrix generation network, and form the first incremental network model with the second network model processed by the first weight matrix and the first feature extraction network and the first weight matrix generation network, and train the first feature extraction network, the first weight matrix generation network and the second classifier of the first incremental network model through the second behavior data set to obtain the second feature extraction network, the second weight matrix generation network and the third classifier.

[0063] In a specific embodiment, step S3 specifically includes:

[0064] S31, inputting the second line data set into the first feature extraction network to extract incremental data features;

[0065] S32, inputting the incremental data features into a first weight matrix generation network to obtain a first weight matrix;

[0066] S33: Multiply the parameters of the second network model by the first weight matrix to obtain the second network model processed by the first weight matrix.

[0067] S34, input the second behavior data set into the second network model processed by the first weight matrix, and output the classification result through the second classifier; use gradient descent and back propagation to train the first feature extraction network, the first weight matrix generation network and the second classifier to obtain the second feature extraction network, the second weight matrix generation network and the third classifier.

[0068] In a specific embodiment, the first weight matrix R=M FL (F ex (x)), the first incremental network model can be expressed as F LSTM (x)×M FL (F ex(x)). During this training process, only the first feature extraction network, the first weight matrix generation network, and the second classifier are trained. The second network model is not trained. During this process, only the parameters of the second network model are matrix multiplied by the first weight matrix. The parameters in the second network model do not change.

[0069] S4: The second network model and the third classifier are trained using the second behavior data set to obtain a third network model and a fourth classifier.

[0070] In a specific embodiment, in step S4, the second network model in the first incremental network model is trained using the second behavior data set. This training process only updates the second network model and the third classifier, and does not update the second feature extraction network and the second weight matrix generation network.

[0071] S5, based on the first behavior data set and the second behavior data set, obtain a third behavior data set, repeat the process of step S3 for the third behavior data set to obtain a second incremental network model, and use the third behavior data set to train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute the target network model.

[0072] In a specific embodiment, step S5 obtains the third behavior data set based on the first behavior data set and the second behavior data set, specifically including: extracting (N1+N2) categories of behavior data from the first behavior data set and the second behavior data set as the third behavior data set, and each category of behavior data in the third behavior data set has K items, where K≤min{K1, K2}. That is, D all ={x i，j |1≤i≤(N new +N old ),1≤j≤K}.

[0073] The process of training the second feature extraction network, the second weight matrix generation network, and the fourth classifier of the second incremental network model using the third behavior data set is the same as step S3. Only the second feature extraction network, the second weight matrix generation network, and the fourth classifier are trained and updated, and the third network model is not updated. Step S5 specifically includes:

[0074] S51, inputting the third line data set into the second feature extraction network to extract incremental data features;

[0075] S52, inputting the incremental data features into a second weight matrix generation network to obtain a second weight matrix;

[0076] S53, multiplying the parameters of the third network model by the second weight matrix to obtain the third network model processed by the second weight matrix;

[0077] S54, input the third behavior data set into the third network model after being processed by the second weight matrix, and output the classification result through the third classifier; use gradient descent and back propagation to train the second feature extraction network, the second weight matrix generation network and the fourth classifier to obtain the third feature extraction network, the third weight matrix generation network and the fifth classifier.

[0078] The third feature extraction network, the third weight matrix generation network, the third network model, and the fifth classifier constitute a target network model. The target network model can effectively identify all behavior categories that appear in the first behavior dataset and the second behavior dataset.

[0079] Furthermore, the third behavior data set can be used as the first behavior training set, and the small sample behavior data of the newly added category can be used as the second training set to repeat steps S3-S5 to obtain the final target network model.

[0080] The trained target network model can recognize more categories of behavioral data, reuse the knowledge learned by the network model, avoid catastrophic forgetting problems, and avoid the requirement to use a large amount of new categories of behavioral data.

[0081] Further references Figure 5 As an implementation of the methods shown in the above figures, this application provides an embodiment of a small sample behavior recognition and classification incremental learning device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0082] This embodiment of the present application provides a small sample behavior recognition and classification incremental learning device, including:

[0083] Initial model construction module 1 is configured to construct a first network model and a first feature extraction network based on an LSTM structure, a first classifier based on a fully connected layer, and a first weight matrix generation network, wherein the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network;

[0084] A first training module 2 is configured to obtain a first behavior data set and a second behavior data set with different categories, and train the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier;

[0085] The second training module 3 is configured to input the second behavior data set into the first feature extraction network, and obtain the first weight matrix from the output of the first weight matrix generation network, form a first incremental network model with the second network model processed by the first weight matrix, the first feature extraction network and the first weight matrix generation network, and train the first feature extraction network, the first weight matrix generation network and the second classifier of the first incremental network model using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network and a third classifier;

[0086] A third training module 4 is configured to train the second network model and the third classifier using the second behavior data set to obtain a third network model and a fourth classifier;

[0087] The fourth training module 5 is configured to obtain a third behavior data set based on the first behavior data set and the second behavior data set, repeat the process of the second training module on the third behavior data set to obtain a second incremental network model, and train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model through the third behavior data set to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute the target network model.

[0088] Reference below Figure 6 , which shows an electronic device (eg Figure 1 A schematic structural diagram of a computer device 600 (a server or terminal device as shown). Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0089] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 603 or the program loaded from the storage part 609 to the random access memory (RAM) 604. Various programs and data required for the operation of the device 600 are also stored in the RAM 604. The CPU 601, GPU 602, ROM 603 and RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0090] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, and the like; an output section 608 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 609 including a hard disk and the like; and a communication section 610 including a network interface card such as a LAN card or a modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 611 as needed, so that a computer program read therefrom can be installed into the storage section 609 as needed.

[0091] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 610, and / or installed from a removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present application are executed.

[0092] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable medium, or any combination of the two. Computer-readable media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses, or components, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0093] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments described in this application may be implemented in software or hardware, and may also be set in a processor.

[0096] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: constructs a first network model and a first feature extraction network based on the LSTM structure, and a first classifier and a first weight matrix generation network based on a fully connected layer, the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network; obtains a first behavior data set and a second behavior data set with different categories, trains the first network model and the first classifier through the first behavior data set, and obtains a second network model and a second classifier; inputs the second behavior data set into the first feature extraction network, and obtains the first weight matrix from the output of the first weight matrix generation network, and combines the second network model processed by the first weight matrix with the first feature extraction network and the first weight matrix generation network to form a first incremental network model. , the first feature extraction network, the first weight matrix generation network and the second classifier of the first incremental network model are trained by the second behavior data set to obtain the second feature extraction network, the second weight matrix generation network and the third classifier; the second network model and the third classifier are trained by the second behavior data set to obtain the third network model and the fourth classifier; the third behavior data set is obtained based on the first behavior data set and the second behavior data set, and the above process is repeated for the third behavior data set to obtain the second incremental network model, and the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model are trained by the third behavior data set to obtain the third feature extraction network, the third weight matrix generation network and the fifth classifier, and the third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute the target network model.

[0097] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A small sample behavior recognition and classification incremental learning method, characterized by: The following steps are involved: S1, constructing a first network model and a first feature extraction network based on an LSTM structure, and a first classifier and a first weight matrix generation network based on a fully connected layer, wherein the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network; S2, obtaining a first behavior data set and a second behavior data set of different categories, where the first behavior data set includes behavior data of N1 categories, and the second behavior data set includes behavior data of N2 categories, and training the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier; S3, inputting the second behavior data set into the first feature extraction network to extract incremental data features; Inputting the incremental data features into the first weight matrix generation network to obtain the first weight matrix, forming a first incremental network model with the second network model processed by the first weight matrix, the first feature extraction network, and the first weight matrix generation network, and training the first feature extraction network, the first weight matrix generation network, and the second classifier of the first incremental network model using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network, and a third classifier; S4: training the second network model and the third classifier using the second behavior data set to obtain a third network model and a fourth classifier; S5. Obtain a third behavior data set based on the first behavior data set and the second behavior data set, repeat the process of step S3 for the third behavior data set to obtain a second incremental network model, and train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model through the third behavior data set to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute a target network model, and the target network model is used to identify different categories of behavior data to obtain corresponding behavior categories.

2. The incremental learning method for small sample behavior recognition and classification according to claim 1 is characterized in that: There are K1 pieces of behavior data for each category in the first behavior data set; there are K2 pieces of behavior data for each category in the second behavior data set.

3. The incremental learning method for small sample behavior recognition and classification according to claim 2 is characterized in that: In step S5, obtaining a third behavior data set based on the first behavior data set and the second behavior data set specifically includes: extracting (N1+N2) categories of behavior data from the first behavior data set and the second behavior data set as the third behavior data set, and each category in the third behavior data set has K pieces of behavior data, where K≤min{K1, K2}.

4. The incremental learning method for small sample behavior recognition and classification according to claim 1 is characterized in that The second network model processed by the first weight matrix in step S3 is specifically obtained by multiplying all parameters of the second network model by the first weight matrix.

5. The incremental learning method for small sample behavior recognition and classification according to claim 1 is characterized in that: In step S3, the first feature extraction network, the first weight matrix generation network, and the second classifier of the first incremental network model are trained using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network, and a third classifier, specifically including: The second behavior data set is input into the second network model of the first incremental network model after being processed by the first weight matrix, and the classification result is output through the second classifier; the first feature extraction network, the first weight matrix generation network and the second classifier are trained by gradient descent and back propagation to obtain a second feature extraction network, a second weight matrix generation network and a third classifier.

6. The incremental learning method for small sample behavior recognition and classification according to claim 1 is characterized in that: Also includes: The third behavior data set is used as the first behavior training set, and the small sample behavior data of the newly added category is used as the second behavior training set, and steps S3-S5 are repeated to obtain the final target network model.

7. A small sample behavior recognition and classification incremental learning device, characterized by: include: An initial model construction module is configured to construct a first network model and a first feature extraction network based on an LSTM structure, a first classifier based on a fully connected layer, and a first weight matrix generation network, wherein the first network model is connected to the first classifier, and the first feature extraction network is connected to the first weight matrix generation network; A first training module is configured to obtain a first behavior data set and a second behavior data set of different categories, the first behavior data set including behavior data of N1 categories, and the second behavior data set including behavior data of N2 categories, and train the first network model and the first classifier using the first behavior data set to obtain a second network model and a second classifier; A second training module is configured to input the second behavior data set into the first feature extraction network to extract incremental data features; Inputting the incremental data features into the first weight matrix generation network to obtain the first weight matrix, forming a first incremental network model with the second network model processed by the first weight matrix, the first feature extraction network, and the first weight matrix generation network, and training the first feature extraction network, the first weight matrix generation network, and the second classifier of the first incremental network model using the second behavior data set to obtain a second feature extraction network, a second weight matrix generation network, and a third classifier; a third training module, configured to train the second network model and the third classifier using the second behavior data set to obtain a third network model and a fourth classifier; The fourth training module is configured to obtain a third behavior data set based on the first behavior data set and the second behavior data set, repeat the process of the second training module on the third behavior data set to obtain a second incremental network model, and train the second feature extraction network, the second weight matrix generation network and the fourth classifier in the second incremental network model through the third behavior data set to obtain a third feature extraction network, a third weight matrix generation network and a fifth classifier. The third feature extraction network, the third weight matrix generation network, the third network model and the fifth classifier constitute a target network model, and the target network model is used to identify different categories of behavior data to obtain corresponding behavior categories.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images