A method and device for establishing a robust incremental behavior recognition model

Abstract

The present invention discloses a method and device for establishing a robust incremental behavior recognition model. First, the present invention constructs a class-incremental behavior recognition model; performs online augmentation on the samples; then uses the online-augmented training data set to train the class-incremental behavior recognition model. During training, if the training samples belong to the old classes, positive-correlated soft labels are generated according to the corresponding labels for self-correction loss calculation; then the class-incremental behavior recognition model with the highest recognition accuracy is selected through the validation data set and saved; finally, the saved model is tested using the test data set to obtain the behavior recognition accuracy and forgetting rate of the current class, and the model is updated according to the incremental classes; until the training of all classes is completed, a trained class-incremental behavior recognition model is obtained. The present invention has high robustness and good performance.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to a method and device for establishing a robust incremental behavior recognition model. Background Art

[0002] Behavior recognition is an important application of computer vision and is widely used in fields such as human-computer interaction, autonomous driving, intelligent monitoring, and healthcare. In the field of medical monitoring, this technology can be used to achieve real-time monitoring of patients to ensure that patients can receive timely treatment and assistance.

[0003] Most of the existing work on human behavior recognition focuses on fixed categories. However, in actual application scenarios, new video categories are constantly emerging. For example, on video platforms such as Douyin, millions of short videos are uploaded by users every day, and some of them belong to new categories or have new labels. Therefore, the behavior recognition model needs to adapt to new categories. However, when the existing behavior recognition model faces incremental categories, "catastrophic forgetting" often occurs, that is, when training the model on a new dataset, the knowledge learned on the old data will be forgotten.

[0004] Incremental learning (also known as continuous learning) can imitate humans' ability to continuously process continuous information flows in the real world, retaining and even integrating and optimizing old knowledge while absorbing new knowledge. The current main research content of incremental learning is image classification, but it has not been fully explored in the field of video behavior recognition. Video data has an additional time dimension compared to images, with a larger sample size and higher computational complexity. Therefore, combining behavior recognition with incremental learning can greatly improve the practicality of this technology in production and life and has very broad application prospects. Summary of the Invention

[0005] Object of the Invention: In view of the problems existing in the prior art, the present invention provides a method and device for establishing a robust incremental behavior recognition model based on video learning.

[0006] Technical Solution: The method for establishing a robust incremental behavior recognition model described in the present invention includes:

[0007] Step S1: Divide a dataset including a number of video samples into a training dataset, a validation dataset, and a test dataset, and determine the training order of samples of different categories in the dataset;

[0008] Step S2: Construct a class-incremental behavior recognition model;

[0009] Step S3: Obtain the category samples to be trained according to the training order, and perform online augmentation on the samples;

[0010] Step S4: Train the class-incremental behavior recognition model using the online-augmented training dataset. During training, if the training sample belongs to an old class, generate a positively correlated soft label according to the corresponding label for self-correction loss calculation. If the training sample belongs to a new class, select several representative samples from the samples belonging to the old class, and select several key frames from the representative samples. Then, form a new-class training dataset sample by combining these key frames with the samples belonging to the new class for training. The old class refers to the classes that have been recognized previously, and the new class refers to the classes that have not been recognized yet.

[0011] Step S5: Screen out the class-incremental behavior recognition model with the highest recognition accuracy through the validation dataset and save it.

[0012] Step S6: Test the saved model using the test dataset, obtain the behavior recognition accuracy and forgetting rate of the current class, and update the model according to the incremental classes.

[0013] Step S7: Return to execute Step S3 until the training of all classes is completed, and obtain the trained class-incremental behavior recognition model.

[0014] Further, in Step S1, the method for determining the learning order of samples of different classes in the dataset is as follows: Select any class of video samples of a determined class in the dataset as the base class for the first training, and the video samples of the remaining classes as the incremental classes for subsequent training.

[0015] Further, in Step S2, the class-incremental behavior recognition model includes a feature extractor and a classifier. Among them, the feature extractor is used to extract features from the input samples, specifically a pre-trained network, and the classifier is used to classify the features extracted by the feature extractor.

[0016] Further, in Step S3, the method of online augmentation is as follows:

[0017] Step S3-1: If the current stage is the base-class training stage, crop the video frames of the sample along the time dimension with the same size, and the cropped spatial resolution size is H×W, where H and W are the height and width of the spatial resolution respectively. If the current stage belongs to the incremental-class training stage, crop the video frames of the sample along the time dimension with the same size, and the cropped spatial resolution size is (aH)×(aW), 0<a≤1. Among them, the training stage for the base class is the base-class training stage, and the training stage for the incremental class is the incremental training stage.

[0018] Step S3-2: Randomly flip the video frames of the sample horizontally along the time dimension with the same degree.

[0019] Further, in step S4, the method for training the class-incremental behavior recognition model using the online-augmented training dataset includes:

[0020] Step S4-1: Set the training parameters. If the current stage is the base training stage, randomly initialize the class-incremental behavior recognition model. If it is the incremental training stage, directly initialize it with the network parameters obtained from the previous training.

[0021] Step S4-2: If the current sample belongs to a new class, select several representative samples from the samples belonging to the old classes, and select several key frames from the representative samples. Combine these key frames with the samples belonging to the new class to form the samples of the new class training dataset. Otherwise, execute step S4-3.

[0022] Step S4-3: After evenly segmenting the video of the sample, randomly sample each segment, and the feature extractor extracts features from the sampled video frames.

[0023] Step S4-4: The classifier classifies the features extracted in step S4-3.

[0024] Step S4-5: Calculate the loss using the cross-entropy loss.

[0025] Step S4-6: If the current sample belongs to the old class, generate a positively correlated soft label according to the corresponding label for self-correction loss calculation. Otherwise, directly execute step S4-7.

[0026] Step S4-7: Combine the losses calculated in step S4-5 and step S4-6 to optimize the network parameters.

[0027] Further, in step S4-6, the method for generating a positively correlated soft label according to the corresponding label for self-correction loss calculation includes:

[0028] Generate a positively correlated soft label according to the corresponding label using the following formula:

[0029]

[0030] In the formula, represents the soft label of sample x, θ t represents the label of sample x, α ∈ (0, 1] is the regularization factor, is the calibration input with the same dimension as The value range of is between 0 and 1, and t represents the current iteration number;

[0031] Calculate the self-correction loss according to the soft label:

[0032]

[0033] In the formula, represents the self-correction loss, O t-1 represents a sample belonging to the old class, o(x) represents the output predicted by the network for the old-class sample, and σ(·) represents the sigmoid operation.

[0034] Furthermore, in the step S4-2, the representative samples are selected by using the herding algorithm.

[0035] Furthermore, in the step S4-2, the method for selecting the key frames is as follows:

[0036] Calculate the difference degree value CY(i) of each frame for the representative samples according to the following formula. If CY(i) ≥ β, then take the current frame as the key frame, where β is a given threshold:

[0037]

[0038] In the formula, CY(i) represents the difference degree value of the i-th frame, γ is a hyperparameter, 0 < γ ≤ 1, j represents the index of the next frame after the previous key frame, f(·) is a characterization function, V · represents the ·-th frame, and T represents the total number of frames of the video to which the current frame belongs.

[0039] Furthermore, in the step S6, the method for updating the model according to the incremental task categories is specifically as follows:

[0040] Use the parameters of the classifier in the previous training stage to update the weights and biases of the old-class classification head in the classifier of the next incremental training stage, and increase the dimension of the classifier according to the number of subsequent incremental categories.

[0041] The robust incremental behavior recognition model establishment device of the present invention includes a processor and a computer program stored on a memory and executable on the processor. When the processor executes the program, the above method is implemented.

[0042] Beneficial effects: Compared with the prior art, the present invention has the remarkable advantages that: the present invention performs incremental behavior recognition based on videos, effectively avoids the distillation strategy during knowledge reproduction, reduces the computational amount while enhancing the incremental performance of the network, and achieves very good performance in terms of accuracy, forgetting rate, and average accuracy in different incremental stages; the soft label generation is stable and accurate, which can effectively avoid the interference of incorrect predictions of the teacher network on the incremental network, and further effectively guide the teacher network to review old knowledge through the self-correction loss function. Brief Description of the Drawings

[0043] Figure 1 is a schematic flowchart of the robust incremental behavior recognition model establishment method provided by the present invention;

[0044] Figure 2 It is a schematic diagram of the training process of the incremental behavior recognition model of the present invention;

[0045] Figure 3 It is a schematic structural diagram of the robust incremental behavior recognition model establishment device provided by the present invention. Specific embodiments

[0046] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0047] Embodiment 1

[0048] This embodiment provides a method for establishing a robust incremental behavior recognition model, as Figure 1 shown, which specifically includes the following steps:

[0049] Step S1: Divide the data set including a number of video samples into a training data set, a validation data set, and a test data set, and determine the learning order of different category samples in the data set.

[0050] Among them, select any one type of video sample of a determined category in the data set as the base class for the first training, and the remaining category of video samples as the incremental class for subsequent training. After the training of the samples of one category is completed, the training of the samples of the next category is carried out. The training stage for the base class is called the base class training stage, and the training stage for the incremental class is called the incremental training stage.

[0051] Step S2: Construct a class incremental behavior recognition model.

[0052] As Figure 2 shown, the class incremental behavior recognition model includes a feature extractor and a classifier. Among them, the feature extractor is used to extract features from the input samples, specifically a pre-trained network, such as the first 10 layers of the VGG16 network, and the classifier is used to classify the features extracted by the feature extractor, such as a fully connected layer.

[0053] Step S3: Obtain the category samples to be trained according to the training order, and perform online augmentation on the samples.

[0054] Among them, the method of online augmentation is specifically as follows:

[0055] Step S3-1: If the current stage is the base class training stage, the video frames of the sample are cropped identically along the time dimension with a cropped spatial resolution size of H×W, where H and W are the height and width of the spatial resolution respectively; if the current stage belongs to the incremental class training stage, the video frames of the sample are cropped identically along the time dimension with a cropped spatial resolution size of (aH)×(aW), where 0 < a ≤ 1.

[0056] Step S3-2: Randomly flip the video frames of the sample horizontally along the time dimension with a probability of 0.5.

[0057] Step S4: Train the class incremental behavior recognition model using the online augmented training dataset.

[0058] The training method includes:

[0059] Step S4-1: Set the training parameters. If the current stage is the base training stage, randomly initialize the class incremental behavior recognition model; if it is the incremental training stage, directly initialize it with the network parameters obtained from the previous training. The training parameters include necessary parameters such as the number of iterations, learning rate, and the number of saved video frame samples of the old classes.

[0060] Step S4-2: If the current sample belongs to a new class, select several representative samples from the samples belonging to the old classes using the herding algorithm, and select several key frames from the representative samples. The method for selecting key frames is as follows:

[0061] Calculate the difference degree value CY(i) of each frame of the representative samples according to the following formula. If CY(i) ≥ β, then take the current frame as a key frame, where β is a given threshold:

[0062]

[0063] In the formula, CY(i) represents the difference degree value of the i-th frame, β is a hyperparameter, 0 < γ ≤ 1, j represents the index of the frame after the previous key frame, f(·) is a characterization function, V· represents the ·-th frame, and T represents the total number of frames of the video to which the current frame belongs.

[0064] Construct the samples of the new class training dataset with these key frames and the samples belonging to the new class; otherwise, execute Step S4-3.

[0065] Step S4-3: After evenly segmenting the sample video, randomly sample each segment, and the feature extractor extracts features from the sampled video frames.

[0066] Step S4-4: The classifier classifies the features extracted in Step S4-3.

[0067] Step S4-5: Calculate the loss using cross-entropy loss.

[0068] Step S4-6: If the current sample belongs to the old class, generate a positively correlated soft label according to the corresponding label for self-correction loss calculation. Specifically: Generate a positively correlated soft label according to the corresponding label using the following formula:

[0069]

[0070] In the formula, represents the soft label of sample x, θ t represents the label of sample x, α∈(0,1] is the regularization factor, is the calibration input with the same dimension as , ranges from 0 to 1, t represents the current iteration number; the label of the i-th sample x i in video v is represented as a one-hot vector where θ v,c ==1, θ v,m ==0|[[]]END]] m∈[1,C],m≠c , representing that the current sample belongs to class c, and C represents the number of old class categories.

[0071] Perform self-correction loss calculation based on the soft label:

[0072]

[0073] In the formula, represents the self-correction loss, O t-1 represents the sample belonging to the old class, o(x) represents the output predicted by the network for the old class sample, and σ(·) represents the sigmoid operation.

[0074] Otherwise, directly execute Step S4-7;

[0075] Step S4-7: Combine the losses calculated in Step S4-5 and Step S4-6 to optimize the network parameters.

[0076] Step S5: Screen out the class incremental behavior recognition model with the highest recognition accuracy through the validation dataset and save it.

[0077] Step S6: Test the saved model using the test dataset, obtain the behavior recognition accuracy and forgetting rate of the current class, and update the model according to the incremental class.

[0078] Step S7: Return to execute Step S3 until the training of all classes is completed, and obtain the trained class incremental behavior recognition model.

[0079] Among them, the method for updating the model according to the incremental task category is specifically as follows: using the parameters of the classifier in the previous training stage to update the weights and biases of the old-class classification head in the classifier in the next incremental training stage, and increasing the dimension of the classifier according to the number of subsequent incremental classes.

[0080] Embodiment 2

[0081] Figure 3 FIG. 7 is a schematic structural diagram of a device provided by an embodiment of the present invention. The embodiment of the present invention provides services for implementing the method of Embodiment 1 above, and can configure a computing device for the method in Embodiment 1 above. Figure 3 FIG. 9 shows a block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present invention. Figure 3 The shown device 12 is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0082] As Figure 3 shown, the device 12 is presented in the form of a general-purpose computing device. The components of the device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components including the system memory 28 and the processing unit 16.

[0083] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0084] The device 12 typically includes a variety of computer system readable media. These media can be any available media accessible by the device 12, including volatile and non-volatile media, removable and non-removable media.

[0085] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be used for reading and writing non-removable, non-volatile magnetic media ( Figure 3 not shown, commonly referred to as a "hard disk drive"). Although Figure 3Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) can be provided. In these cases, each drive can be connected to the bus 18 through one or more data medium interfaces. The memory 28 may include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0086] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in the memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment. The program modules 42 generally perform the functions and / or methods in the embodiments described in the present invention.

[0087] The device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the device 12, and / or communicate with any device that enables the device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the device 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As Figure 3 shown, the network adapter 20 communicates with other modules of the device 12 through the bus 18. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0088] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the method provided in the first embodiment of the present invention.

Claims

1. A method for establishing a robust incremental behavior recognition model, characterized in that Including: Step S1: Divide a dataset including a number of video samples into a training dataset, a validation dataset, and a test dataset, and determine the training order of samples of different categories in the dataset; Step S2: Construct a class-incremental behavior recognition model; Step S3: Obtain the category samples to be trained according to the training order, and perform online augmentation on the samples; Step S4: Use the online-augmented training dataset to train the class-incremental behavior recognition model. During training, if the training sample belongs to an old class, generate a positively correlated soft label according to the corresponding label for self-correction loss calculation. If the training sample belongs to a new class, select several representative samples from the samples belonging to the old class, and select several key frames from the representative samples. Combine these key frames with the samples belonging to the new class to form the samples of the new class training dataset for training; the old class is the category that has been recognized before, and the new class is the category that has not been recognized yet; Step S5: Screen out and save the class-incremental behavior recognition model with the highest recognition accuracy through the validation dataset; Step S6: Use the test dataset to test the saved model, obtain the behavior recognition accuracy and forgetting rate of the current category, and update the model according to the incremental category; Step S7: Return to execute Step S3 until the training of all classes is completed, and obtain the trained class-incremental behavior recognition model.

2. The method for establishing a robust incremental behavior recognition model according to claim 1, wherein: In Step S1, the determination of the learning order of samples of different categories in the dataset is specifically as follows: Select any one type of video sample of a determined category in the dataset as the base class for the first training, and the video samples of the remaining categories as the incremental classes for subsequent training.

3. The method for establishing a robust incremental behavior recognition model according to claim 1, wherein: In Step S2, the class-incremental behavior recognition model includes a feature extractor and a classifier. Among them, the feature extractor is used to extract features from the input samples, specifically a pre-trained network, and the classifier is used to classify the features extracted by the feature extractor.

4. The method for establishing a robust incremental behavior recognition model according to claim 1, characterized in that: In Step S3, the method of online augmentation is specifically as follows: Step S3-1: If the current stage is the base class training stage, crop the video frames of the sample along the time dimension with the same size, and the cropped spatial resolution size is H×W, where H and W are the height and width of the spatial resolution respectively; if the current stage belongs to the incremental class training stage, crop the video frames of the sample along the time dimension with the same size, and the cropped spatial resolution size is (aH)×(aW), 0<a≤1; among them, the training stage for the base class is the base class training stage, and the training stage for the incremental class is the incremental training stage; Step S3-2: Randomly flip the video frames of the sample horizontally along the time dimension.

5. The method for establishing a robust incremental behavior recognition model according to claim 1, characterized in that: In Step S4, the method of using the online-augmented training dataset to train the class-incremental behavior recognition model includes: Step S4-1: Set the training parameters. If the current stage is the base training stage, randomly initialize the class-incremental behavior recognition model. If it is the incremental training stage, directly initialize it with the network parameters obtained from the previous training; Step S4-2: If the current sample belongs to a new class, select several representative samples from the samples belonging to the old class, and select several key frames from the representative samples. Combine these key frames with the samples belonging to the new class to form the samples of the new class training data set; otherwise, execute Step S4-3; Step S4-3: After evenly segmenting the video of the sample, randomly sample each segment, and the feature extractor extracts features from the sampled video frames; Step S4-4: The classifier classifies the features extracted in Step S4-3; Step S4-5: Calculate the loss using the cross-entropy loss; Step S4-6: If the current sample belongs to the old class, generate a positively correlated soft label according to the corresponding label for self-correction loss calculation; otherwise, directly execute Step S4-7; Step S4-7: Combine the losses calculated in Step S4-5 and Step S4-6 to optimize the network parameters.

6. The method for establishing a robust incremental behavior recognition model according to claim 5, characterized in that: In the said Step S4-6, the method of generating a positively correlated soft label according to the corresponding label for self-correction loss calculation includes: Use the following formula to generate a positively correlated soft label according to the corresponding label: In the formula, represents the soft label of sample x, and θ t represents the label of sample x. α ∈ (0, 1] is the regularization factor. is the calibration input with the same dimension as . The value range of is between 0 and 1, and t represents the current iteration number. Calculate the self-correction loss according to the soft label: In the formula, represents the self-correction loss, O t-1 represents the samples belonging to the old class, o(x) represents the output predicted by the network for the old-class samples, and σ(·) represents the sigmoid operation.

7. The method for establishing a robust incremental behavior recognition model according to claim 5, characterized in that: In the said Step S4-2, the representative samples are selected by using the herding algorithm.

8. The method for establishing a robust incremental behavior recognition model according to claim 5, wherein: In the said Step S4-2, the method of selecting the key frames is: Calculate the difference degree value CY(i) of each frame of the representative sample according to the following formula. If CY(i)≥β, then take the current frame as the key frame, where β is a given threshold: In the formula, CY(i) represents the difference degree value of the i-th frame, γ is a hyperparameter, 0<γ≤1, j represents the index of the next frame of the previous key frame, f(·) is a characterization function, V· represents the ·-th frame, and T represents the total number of frames of the video to which the current frame belongs.

9. The method for establishing a robust incremental behavior recognition model according to claim 1, characterized in that: In the said Step S6, the method of updating the model according to the incremental task category is specifically: Use the parameters of the classifier in the previous training stage to update the weights and biases of the old class classification head in the classifier in the next incremental training stage, and increase the dimension of the classifier according to the number of subsequent incremental categories.

10. A robust incremental behavior recognition model establishment device, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the said processor executes the said program, it implements the method described in any one of claims 1-9.

Images