Multi-task behavior recognition model training method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a multi-task behavior recognition model training method, device, equipment and storage medium, wherein the method comprises: obtaining a sample data set of N tasks to be recognized, wherein the sample data set comprises at least one of virtual data, data collected by a camera device and crawled data from the Internet, and N is an integer greater than or equal to 1; using the sample data sets of the N tasks to be recognized to train a constructed behavior recognition model to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model comprises a first backbone network and N first classification networks, the multi-task behavior recognition model comprises a multi-task backbone network and N second classification networks, the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be recognized and are obtained by training the N first classification networks.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of behavior recognition, and are related to but not limited to a multi-task behavior recognition model training method, device, equipment and storage medium. Background Art

[0002] In the actual application of multi-task abnormal behavior recognition, when there is a large domain difference between different tasks, the results of multi-task parallel learning inhibit each other, which limits the accuracy of multi-task learning. Training a multi-task abnormal behavior recognition model requires a large amount of data in the training task set. There needs to be sufficient data support between tasks, and the amount of data between classes needs to be balanced, otherwise it is impossible to train a robust discriminator. The current abnormal task data set has limited ways to obtain it, and the number cannot meet the training requirements. Summary of the invention

[0003] In view of this, embodiments of the present application provide a multi-task behavior recognition model training method, apparatus, device and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a multi-task behavior recognition model training method, comprising: obtaining a sample data set of N tasks to be recognized, wherein the sample data set includes at least one of virtual data, data collected by a camera device, and data crawled from the Internet, and N is an integer greater than or equal to 1; using the sample data sets of the N tasks to be recognized to train a constructed behavior recognition model to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, the multi-task behavior recognition model includes a multi-task backbone network and N second classification networks, the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be recognized and are obtained by training the N first classification networks.

[0006] In some embodiments, the constructed behavior recognition model is obtained by the following method: obtaining the first backbone network, wherein the first backbone network is a backbone network obtained by training a preset data set; adding the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model.

[0007] In this way, first obtain the first backbone network trained by the preset data set, and then add the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model. In this way, the behavior recognition model obtained by combining a backbone network with multiple classification networks can be trained using a sample data set to obtain a behavior recognition model that can perform multi-task recognition. In this way, the model architecture can realize multiple tasks on a single model, greatly reducing the video memory consumption caused by adding recognition tasks.

[0008] In some embodiments, the behavior recognition model constructed by training with the sample data sets of the N tasks to be identified to obtain a multi-task behavior recognition model includes: training the first backbone network in the constructed behavior recognition model with the sample data sets of the N tasks to be identified to obtain the multi-task backbone network; training the first behavior recognition model with the sample data sets of the N tasks to be identified to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0009] In this way, by first training a fixed backbone network and then separately training the classification networks corresponding to different tasks to be identified, a behavior recognition model suitable for multiple tasks can be obtained.

[0010] In some embodiments, the use of the sample data sets of the N tasks to be identified to train the first backbone network in the constructed behavior recognition model to obtain the multi-task backbone network includes: using the sample data sets of each of the N tasks to be identified to train N first teacher models respectively to obtain N second teacher models corresponding one by one to the N tasks to be identified, wherein each of the first teacher models includes the first backbone network, a first classification network, a variational autoencoder and a discriminator; using N second teacher models to train the first backbone network to obtain the multi-task backbone network.

[0011] In this way, the first backbone network is trained using N second teacher models corresponding to N different tasks to be identified. The multi-task backbone network obtained through knowledge distillation can extract features of different tasks to be identified on one backbone network, effectively improving the feature extraction accuracy of the backbone network.

[0012] In some embodiments, the method of using N second teacher models to train the first backbone network to obtain the multi-task backbone network includes: adjusting the number of batch normalization layers in the first backbone network to obtain a second backbone network including N batch normalization layers, wherein each batch normalization layer corresponds to a task to be identified; obtaining feature vectors extracted from all samples in each of the N second teacher models; and using the feature vectors extracted from all samples in each of the second teacher models as target features to train the second backbone network to obtain the multi-task backbone network.

[0013] In this way, the feature vectors extracted from all samples in each second teacher model are used as target features to train the second backbone network, so that the trained multi-task backbone network can improve the accuracy of feature extraction.

[0014] In some embodiments, the method further includes: fusing N batch normalization layers in the multi-task backbone network to obtain a third backbone network; training the third backbone network using sample data sets of the N tasks to be identified to adjust the parameters of the batch normalization layer in the third backbone network to obtain a multi-task backbone network that has been adjusted.

[0015] In this way, the problem of inconsistent distribution of different tasks is solved through different batch normalization layers. Setting the batch normalization layer can solve the problem of changes in the data distribution of the intermediate layer during training, so as to prevent the gradient from disappearing or exploding and speed up the training. Fix the parameters of the second backbone network except the batch normalization layer, merge N batch normalization layers into one batch normalization layer, and further adjust the second backbone network; finally, through the input of each task, fix the remaining parameters of the multi-task backbone network, and adjust the batch normalization layer separately to solve the problem of inconsistent data distribution.

[0016] In some embodiments, when a new task to be identified is added to the behavior recognition model of the multi-task, the method further includes: obtaining the behavior recognition model of the new task, wherein the behavior recognition model of the new task is obtained by adding a first classification network to the behavior recognition model of the multi-task, and the added first classification network is used to classify the new task to be identified; obtaining a new data set corresponding to the new task to be identified; and training the behavior recognition model of the new task using the new data set to obtain a behavior recognition model of the new task that has completed the training.

[0017] In this way, the basic network structure is that multiple tasks share a multi-task backbone network. By adding a task, a fully connected layer or head network is added without adding a multi-task backbone network. By sharing parameters of the multi-task backbone network among various tasks, the network parameters are greatly reduced.

[0018] In some embodiments, the multi-task behavior recognition model is used to identify abnormal behavior of a single person and / or multiple people; the N tasks to be recognized include at least one of the following: a single person's behavior that endangers one's own safety, illegal behavior; multiple people's behavior that endangers one's own safety, illegal behavior. The method also includes: acquiring an image to be recognized; inputting the image to be recognized into the multi-task behavior recognition model to obtain an abnormal behavior recognition result; and outputting the abnormal behavior recognition result.

[0019] In this way, since urban safety and harmony have always been a very important aspect in the scenario of smart city management. Therefore, when abnormal events occur, such as fighting, elevator falls, kicking and other dangerous actions, it is often necessary to be able to alarm the management personnel in the first time. The embodiment of the present application can identify the urban situation in real time. When dangerous actions occur and cause urban hazards, such as fighting, the system can generate an alarm in the first time, without the need for management personnel to watch the video taken by the camera device 24 hours a day, which greatly saves manpower and material resources and ensures urban safety.

[0020] In a second aspect, an embodiment of the present application provides a multi-task behavior recognition model training device, comprising: a first acquisition module, used to acquire a sample data set of N tasks to be identified, wherein the sample data set includes at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1; a first training module, used to train a constructed behavior recognition model using the sample data sets of the N tasks to be identified to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, the multi-task behavior recognition model includes a multi-task backbone network and N second classification networks, the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be identified, and are obtained by training the N first classification networks.

[0021] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and the processor implements the above method when executing the program.

[0022] In a fourth aspect, an embodiment of the present application provides a storage medium storing executable instructions for causing a processor to execute and implement the above method.

[0023] The embodiments of the present application provide a multi-task behavior recognition model training method, device, equipment and storage medium. First, the method of obtaining the sample data set is multi-source, that is, the sample data set includes at least one of virtual data, data collected by the camera device and crawled data from the Internet. In this way, the sample size of the data set is greatly enriched. The collection and training of multi-source and multi-domain data is helpful for the actual verification of abnormal behavior algorithms and improves the generalization ability of behavior algorithms. Then, the behavior recognition model built by training the sample data sets of N tasks to be identified is obtained to obtain a multi-task behavior recognition model. In this way, the recognition of multiple tasks to be identified can be completed on a single model, which can not only improve the accuracy of multi-task recognition but also greatly reduce the video memory consumption caused by increasing the recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A A schematic diagram of collecting data of a camera device is provided for an embodiment of the present application;

[0025] Figure 1B A schematic diagram of another camera device collecting data is provided for an embodiment of the present application;

[0026] Figure 1C A schematic diagram of crawling data from the Internet is provided for an embodiment of the present application;

[0027] Figure 1D A schematic diagram of generating data based on a virtual game provided in an embodiment of the present application;

[0028] Figure 1E A schematic diagram of another method of generating data based on a virtual game provided in an embodiment of the present application;

[0029] Figure 1F A schematic diagram of a multi-task recognition model training implementation process provided in an embodiment of the present application;

[0030] Figure 2A A schematic diagram of a behavior recognition model provided in an embodiment of the present application;

[0031] Figure 2B A schematic diagram of a teacher model corresponding to each abnormal task obtained through training provided in an embodiment of the present application;

[0032] Figure 2C A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0033] Figure 2D A schematic diagram of the composition structure of another image processing device provided in an embodiment of the present application;

[0034] Figure 3AA schematic diagram of a training process for adding a task to be identified to a multi-task behavior recognition model provided in an embodiment of the present application;

[0035] Figure 3B A multi-task behavior recognition model architecture diagram provided in an embodiment of the present application;

[0036] Figure 3C A schematic diagram of an implementation flow of adding a task to be identified to the multi-task behavior recognition model provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of a multi-task behavior recognition model training implementation process provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of the structure of a multi-task behavior recognition model device provided in an embodiment of the present application;

[0039] Figure 6 A hardware entity schematic diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the invention will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0041] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0042] It should be understood that some embodiments described herein are only used to explain the technical solution of the present application, and are not used to limit the technical scope of the present application.

[0043] This embodiment proposes a multi-task behavior recognition model training method, which is applied to a computer device. The functions implemented by the method can be implemented by calling program codes through a processor in the computer device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the computer device includes at least a processor and a storage medium.

[0044] The present application embodiment provides a multi-task recognition model training implementation flow diagram, such as Figure 1F As shown, the method includes:

[0045] Step S101, obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0046] In some embodiments, the captured data of the camera device may refer to a video or image captured from the camera device, such as Figure 1A As shown, the data collected in the image shows that multiple people are involved in a fight; Figure 1B As shown, the image shows two people climbing over the seat.

[0047] In some embodiments, crawled data from the Internet may refer to data obtained from the Internet, such as Figure 1C As shown, the image shows that two people are fighting. In the implementation process, the method of crawling data can be a crawler algorithm, that is, automatically finding corresponding data on the Internet for keywords.

[0048] In some embodiments, virtual data may be data obtained from an interface of a virtual game, or may refer to data produced using a virtual game engine, such as Figure 1D As shown in the image, a person is climbing a wall; Figure 1E As shown in the figure, the image shows a person jumping. In this way, obtaining sample data sets from different data sources greatly increases the sample size of the data set.

[0049] Step S102, using the sample data sets of the N tasks to be identified to train the constructed behavior recognition model to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, and the multi-task behavior recognition model includes a multi-task backbone network obtained by training the first backbone network and N second classification networks corresponding to the N tasks to be identified obtained by training the N first classification networks.

[0050] In some embodiments, the constructed behavior recognition model may be a behavior recognition model including a first backbone network and N first classification networks.

[0051] The backbone network (BB) is a network used to extract features from input data. It represents a part of the network and is generally used to extract image information at the front end for use by the subsequent network. VGGNet, Residual Neural Network (ResNet), and inception network models are commonly used. Among them, VGGNet is a deep convolutional neural network. These BBs have strong feature extraction capabilities and can load official model parameters trained on large data sets, then connect to your own network and perform fine-tuning.

[0052] The first classification network can be implemented by fully connected layers (FC) or head networks. The fully connected layers play the role of "classifier" in the entire convolutional neural network. The head network has a similar role to the fully connected layers, which also realizes the classification of each category. In the implementation process, the head network can be a lightweight network. According to actual needs, the lightweight network can include vector convolution operation layers and fully connected layers. The fully connected layer plays the role of mapping the learned "distributed feature representation" to the sample label space.

[0053] In actual use, the fully connected layer can be implemented by convolution operations. The behavior recognition model is trained using sample data sets of N tasks to be recognized to obtain a multi-task behavior recognition model.

[0054] Here, the multi-task behavior recognition model includes a multi-task backbone network obtained by training the first backbone network and N second classification networks corresponding to the N tasks to be recognized obtained by training N first classification networks. The second classification networks are obtained by training the corresponding first classification networks again using the sample data sets of the N tasks to be recognized after completing the training of the multi-task backbone network.

[0055] In the embodiment of the present application, first, the method of obtaining the sample data set is multi-source, that is, the sample data set includes at least one of virtual data, data collected by the camera device, and crawled data from the Internet. In this way, the sample size of the data set is greatly enriched. The collection and training of multi-source and multi-domain data is helpful for the actual verification of abnormal behavior algorithms and improves the generalization ability of behavior algorithms. Then, the behavior recognition model built by training the sample data sets of N tasks to be identified is obtained to obtain a multi-task behavior recognition model. In this way, the recognition of multiple tasks to be identified can be completed on a single model, which can not only improve the accuracy of multi-task recognition but also greatly reduce the video memory consumption caused by increasing the recognition tasks.

[0056] The present application provides a method for obtaining a built behavior recognition model, the method comprising:

[0057] Step S111, obtaining the first backbone network, wherein the first backbone network is a backbone network obtained by training a preset data set;

[0058] In some embodiments, the existing data set may use the kinetics-400 data set, which includes four hundred human action categories, each category has at least 400 video clips, each clip is taken from a different Youtube video, and lasts about ten seconds. The action categories of the data set include interactions between people and objects, and interactions between people.

[0059] In some embodiments, the existing dataset also includes mageNet, which is a large visual database for visual object recognition software research, including more than 14 million image URLs that are manually annotated by ImageNet to indicate objects in the image; in at least one million images, bounding boxes are also provided. ImageNet contains more than 20,000 categories.

[0060] Generally speaking, video datasets such as Kinetics-400 are much larger than 2D image datasets, and the data input and output are much higher than images. For example, ImageNet has 1.28 million training images, while the video dataset Kinetics-400 has 63 million training frames, which is about 50 times the former. Using existing datasets such as Kinetics-400, we can complete the pre-training backbone network and obtain the first backbone network.

[0061] Step S112: adding the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model.

[0062] Here, the added N first classification networks may be networks with the same structure and parameters and having classification functions, or may be classification networks obtained through pre-training according to different tasks to be identified.

[0063] In an embodiment of the present application, the first backbone network obtained by training with a preset data set is first obtained, and then the N first classification networks are added to the first backbone network to obtain the constructed behavior recognition model. In this way, the behavior recognition model including a backbone network and multiple classification networks can be trained using a sample data set to obtain a behavior recognition model that can perform multi-task recognition. In this way, the model architecture can realize the recognition of multiple tasks to be recognized on a single model, greatly reducing the video memory consumption caused by adding recognition tasks.

[0064] Figure 2A A schematic diagram of a behavior recognition model provided in an embodiment of the present application, wherein Figure 2A As shown, the constructed behavior recognition model includes a first backbone network, a fully connected layer 1 or a head network 1, a fully connected layer 2 or a head network 2, and a fully connected layer 3 or a head network 3.

[0065] In some embodiments, according to different tasks to be identified, they can be divided into first-class abnormal behavior T1, second-class abnormal behavior T2 and first-class abnormal behavior T3. For example, the three types of abnormal behaviors can correspond to fighting, climbing and falling respectively.

[0066] The first backbone network (BB Share) is a general backbone network used to identify different tasks to be identified, that is, in the initial stage, it can be randomly initialized to obtain Figure 2A The backbone network 1 corresponding to the T1 type of task to be identified, the backbone network 2 corresponding to the T2 type of task to be identified, and the backbone network 3 corresponding to the T3 type of task to be identified are shown; then, a general backbone network is realized by integrating the backbone network 1, the backbone network 2, and the backbone network 3, that is, the network parameters of the backbone network 1, the backbone network 2, and the backbone network 3 are averaged to obtain the network parameters of the general backbone network; finally, the network parameters of the general backbone network are controlled to remain unchanged except for the network parameters of the batch normalization layer (BN), and only the network parameters of the batch normalization layer in the general backbone network can be adjusted to obtain a general backbone network suitable for T1, T2, and T3 tasks. The batch in the batch normalization layer is the batch data, that is, the number of samples in each optimization. The batch normalization layer is usually used after the convolution layer to readjust the data distribution. Setting the batch normalization layer can solve the problem of changes in the data distribution of the intermediate layer during the training process, so as to prevent the gradient from disappearing or exploding and speed up the training speed.

[0067] The fully connected layer 1 is the fully connected layer corresponding to the T1 type of task to be recognized, the fully connected layer 2 is the fully connected layer corresponding to the T2 type of task to be recognized, and the fully connected layer 3 is the fully connected layer corresponding to the T3 type of task to be recognized.

[0068] Head network 1 is the head network corresponding to the T1 type of task to be identified, and is used to classify the T1 type of task to be identified; head network 2 is the head network corresponding to the T2 type of task to be identified, and is used to classify the T2 type of task to be identified; head network 3 is the head network corresponding to the T3 type of task to be identified, and is used to classify the T3 type of task to be identified.

[0069] Here, the network that plays a classification role in the built behavior recognition model can choose to use either the fully connected layer or the head network for each task to be recognized.

[0070] The present application provides a method for implementing multi-task recognition model training, the method comprising:

[0071] Step S201, obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0072] Step S202: training the first backbone network in the constructed behavior recognition model using the sample data sets of the N tasks to be recognized, to obtain the multi-task backbone network;

[0073] The multi-task backbone network here can be a backbone network suitable for multiple different tasks to be identified, so it is necessary to use the sample data sets of the N tasks to be identified to train the first backbone network in the constructed behavior recognition model to obtain a universal, fixed backbone network that can identify different tasks to be identified.

[0074] In some embodiments, Figure 2A As shown, the first backbone network (BB Share) is the first backbone network in the constructed behavior recognition model, and is a general backbone network for identifying different tasks to be identified.

[0075] Step S203: training a first behavior recognition model using the sample data sets of the N tasks to be recognized to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0076] First, N first classification networks are built on the basis of the trained multi-task backbone network to obtain a first behavior recognition model. Then, the sample data sets of N tasks to be recognized are used to train the N first classification networks in the first behavior recognition model that correspond to the tasks to be recognized one by one to obtain a multi-task behavior recognition model.

[0077] Here, the multi-task behavior recognition model includes a multi-task backbone network obtained by training the first backbone network, and N second classification networks corresponding to the N tasks to be recognized, obtained by training N first classification networks.

[0078] In some embodiments, Figure 2A As shown, N is 3, that is, there are 3 tasks to be identified. Here, the first classification network can correspond to the following 3 classification networks, including fully connected layer 1 or head network 1, fully connected layer 2 or head network 2, and fully connected layer 3 or head network 3.

[0079] In an embodiment of the present application, sample data sets from multiple sources are first obtained, and then the first backbone network in the constructed behavior recognition model is trained to obtain a multi-task backbone network. Finally, the first behavior recognition model including the multi-task backbone network and N first classification networks is trained to obtain a multi-task behavior recognition model. In this way, a fixed backbone network is trained first, and then the classification networks corresponding to different tasks to be identified are trained separately, so that a behavior recognition model suitable for multiple tasks can be obtained. The multi-task behavior recognition model greatly reduces the video memory consumption caused by increasing the recognition tasks.

[0080] Figure 2B A schematic diagram of a teacher model corresponding to each abnormal task obtained through training provided in an embodiment of the present application, such as Figure 2BAs shown, the teacher model corresponding to each abnormal task includes a preprocessing module (preprocess) 201, an augmentation processing module (augmentation) 202, a backbone network 203, and a head network H k (Fully connected layer F k ) 204, Variational Auto-Encoder (VAE) 205 and Discriminator (Dis) 206, where:

[0081] The preprocessing module 201 is the processing performed on the input image before feature extraction, segmentation and matching in image analysis. The main purpose of the preprocessing module 201 is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition.

[0082] The enhancement processing module 202 can be used to enhance the useful information in the image. Here, it can be a distortion process, the purpose of which is to improve the visual effect of the image, for the application of a given image. It purposefully emphasizes the overall or local characteristics of the image, makes the originally unclear image clear or emphasizes certain interesting features, expands the difference between the features of different objects in the image, suppresses uninteresting features, so as to improve the image quality, enrich the amount of information, enhance the image interpretation and recognition effect, and meet the needs of certain special analysis.

[0083] Head Network H k (Fully connected layer F k )204 is the classification network in each teacher model; the variational autoencoder 205 can obtain the invariant representation of each domain data after adjusting the different data distributions through the network, and further align the feature space between two different domains through adversarial learning; the discriminator 206 can adaptively measure the difference between two overall distributions, that is, a continuous probability distribution, and is a learnable loss function; classification i is the classification result after the image is input into the teacher model.

[0084] The present invention provides a multi-task recognition model training implementation flow diagram, the method comprising:

[0085] Step S211, obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0086] Step 212: using the sample data set of each of the N tasks to be identified, respectively train N first teacher models to obtain N second teacher models corresponding to the N tasks to be identified, wherein each of the first teacher models includes the first backbone network, a first classification network, a variational autoencoder and a discriminator;

[0087] The first teacher model is trained using various types of multi-source data (data collected by cameras, crawled data from the Internet, and generated data based on virtual games) to obtain the second teacher model.

[0088] Here, when training the first teacher model, we focus on cross-domain (outdoor data, indoor data) training because the actual application data source is relatively rich. Through domain adaptation, the data collected by the camera, the crawled data on the Internet, and the generated data based on virtual games learn consistent behavior representations.

[0089] like Figure 2B As shown, the backbone network 203 is used to extract image features, the head network 204 is used to classify the image features, the variational autoencoder 205 is used to further extract domain information using the image features, and the discriminator 206 is used to make discrimination by obtaining the classification results of the head network and the domain information to adjust the network parameters. After the network parameters are adjusted, an invariant representation of the data of each domain can be obtained, so that the first teacher model can further align the feature space between two different domains through adversarial learning of different data distributions.

[0090] Step S213: using N of the second teacher models to train the first backbone network to obtain the multi-task backbone network;

[0091] Here, the first backbone network is trained using the second teacher models corresponding to N different tasks to be identified, that is, multiple teacher models are used to train a student model. In this way, the best accuracy of each category (task to be identified) on each model can be maintained, so that the generated new model (first backbone network) has better representation ability on each task to be identified.

[0092] Step S214: train a first behavior recognition model using the sample data sets of the N tasks to be recognized to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0093] In an embodiment of the present application, first, when training the first teacher model for each task to be identified, a variational encoder is used to adjust the network to obtain an invariant representation of the data in each domain. The feature space between two different domains is further aligned by adversarial learning of different data distributions. Secondly, the N second teacher models corresponding to N different tasks to be identified are used to train the first backbone network. The multi-task backbone network obtained by knowledge distillation can extract features of different tasks to be identified on a backbone network, effectively improving the feature extraction accuracy of the backbone network. Such a model can perform recognition of multiple tasks to be identified on a single model, greatly reducing the video memory consumption caused by increasing the recognition tasks.

[0094] Figure 2C A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 2C As shown, the image processing device includes a preprocessing module 201, an enhancement processing module 202 and a second backbone network (N batch normalization layers 207), wherein the N batch normalization layers 207 are adjusted (Finetune) using the classification results of the second teacher model (classification i: teacher features).

[0095] Figure 2D A schematic diagram of the structure of another image processing device provided in an embodiment of the present application is shown in FIG. Figure 2D As shown, the image processing device includes a preprocessing module 201, an enhancement processing module 202 and a multi-task backbone network, wherein the multi-task backbone network is integrated with N batch normalization layers 208, wherein the integration of N batch normalization layers 208 is adjusted using the classification results of the second teacher model (classification i: teacher features).

[0096] During the implementation process, Figure 2D The multi-task backbone network shown in Figure 2C The second backbone network shown in is obtained, that is, the multi-task backbone network is obtained by fusing N batch normalization layers. The fusion method can be to average the parameters of the N batch normalization layers and then fine-tune the parameters to obtain the multi-task backbone network.

[0097] The present invention provides a method for implementing a multi-task recognition model training, the method comprising:

[0098] Step S221, obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0099] Step S222: using the sample data set of each of the N tasks to be identified, respectively training N first teacher models to obtain N second teacher models corresponding to the N tasks to be identified, wherein each of the first teacher models includes the first backbone network, a first classification network, a variational autoencoder and a discriminator;

[0100] Step S223: adjusting the number of batch normalization layers in the first backbone network to obtain a second backbone network including N batch normalization layers, wherein each batch normalization layer corresponds to a task to be identified;

[0101] Since the data distribution of each category is inconsistent, the multi-task backbone network can be adjusted to have N batch normalization layers, that is, each task has its own batch normalization layer, and different batch normalization layers can be used to solve the problem of inconsistent distribution of different tasks.

[0102] Step S224, obtaining feature vectors extracted from all samples in each of the N second teacher models;

[0103] Here, all sample feature vectors in each of the N second teacher models are obtained in order to obtain a better multi-task backbone network through knowledge distillation.

[0104] Step S225, using the feature vectors extracted from all samples in each of the second teacher models as target features to train the second backbone network to obtain the multi-task backbone network;

[0105] In the implementation process, the features of the all-in-one model are aligned with the features of a single model through the knowledge distillation method to obtain a better backbone network, that is, the features of the single model are used as target features to guide the training of the backbone network of the all-in-one model to obtain the requirements of consistent features. That is, all the sample feature vectors in each of the second teacher models are used as target features to train the first backbone network in the constructed behavior recognition model. The features of all samples in each category can be obtained through the second teacher model corresponding to each task to be identified.

[0106] The knowledge distillation method can obtain more comprehensive knowledge that makes the target easier to learn, such as features or softer outputs, and separate different feature schemes through the knowledge distillation method. The core idea of ​​knowledge distillation is to transfer knowledge, so as to obtain a small model that is more suitable for reasoning through a trained large model.

[0107] like Figure 2CAs shown, the image is first processed by the preprocessing module 201 and the enhancement processing module 202 to obtain an image that has completed preprocessing and enhancement processing, and then the image is normalized using N batch normalization layers 207. Finally, the N batch normalization layers 207 are adjusted using the classification results of the second teacher model.

[0108] Step S226, fusing the N batch normalization layers in the multi-task backbone network to obtain a third backbone network;

[0109] The parameters except the batch normalization layer in the multi-task backbone network are fixed, and N batch normalization layers can be fused into one batch normalization layer or N batch normalization layers can be fused into M batch normalization layers, where M is less than N. In the implementation process, the fusion method can be to first average the parameters of the N batch normalization layers and then fine-tune the parameters.

[0110] Step S227: training the third backbone network using the sample data sets of the N tasks to be identified, so as to adjust the parameters of the batch normalization layer in the third backbone network, and obtain a multi-task backbone network that has been adjusted;

[0111] like Figure 2D As shown, the image is first processed by the preprocessing module 201 and the enhancement processing module 202 to obtain an image that has completed preprocessing and enhancement processing, and then the image is normalized using the multi-task backbone network 211 that integrates N batch normalization layers. Finally, the classification result of the second teacher model is used to adjust the parameters of the batch normalization layer in the third backbone network to obtain a multi-task backbone network that has completed the adjustment.

[0112] Step S228: Use the sample data sets of the N tasks to be identified to train a first behavior recognition model to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0113] In an embodiment of the present application, first, the multi-task backbone network is adjusted to have N batch normalization layers, and each task has its own batch normalization layer. Different batch normalization layers are used to solve the problem of inconsistent distribution of different tasks. Setting the batch normalization layer can solve the problem of changes in the distribution of intermediate layer data during training, so as to prevent the gradient from disappearing or exploding and speed up the training speed; then, the parameters of the second backbone network except the batch normalization layer are fixed, the N batch normalization layers are fused into one batch normalization layer, and the second backbone network is further adjusted; finally, through the input of each task, the remaining parameters of the multi-task backbone network are fixed, and the batch normalization layer is adjusted separately to solve the problem of inconsistent data distribution.

[0114] Figure 3AA schematic diagram of a training process for adding a task to be identified to a multi-task behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 3A As shown, a model for adding a task to be identified on the multi-task behavior recognition model includes an image pre-processing module 201 and an enhancement processing module 202, a multi-task backbone network 209 and a newly added head network H k (Added fully connected layer F k )210, wherein the classification result of the second teacher model, namely the teacher target (classification i), is used to add a new classification network, namely the fully connected layer (F k ) or head network (H k )210 for adjustment.

[0115] Figure 3B A multi-task behavior recognition model architecture diagram provided in an embodiment of the present application, such as Figure 3B As shown, the multi-task behavior recognition model includes a fixed multi-task backbone network 209 and N second classification networks corresponding one by one to the N tasks to be recognized, namely, a fully connected layer 211 or a head network 212.

[0116] The embodiment of the present application provides a schematic diagram of a process for implementing a new task to be identified in the multi-task behavior recognition model, such as Figure 3C As shown, the method includes:

[0117] Step S301, obtaining a behavior recognition model for a newly added task, wherein the behavior recognition model for the newly added task is obtained by adding a classification network to the behavior recognition model for the multiple tasks, and the added classification network is used to classify the newly added task to be recognized;

[0118] Here, to add a new task of anomaly recognition, we only need to fix the multi-task backbone network and train the newly added fully connected layers or head networks.

[0119] Step S302: Acquire a new data set corresponding to the new task to be identified;

[0120] During the implementation process, the acquisition of new data sets may include data collected by a camera device, data crawled from the Internet, and data generated based on virtual games.

[0121] Step S303: Use the newly added data set to train the behavior recognition model of the newly added task to obtain a behavior recognition model of the newly added task that has completed training.

[0122] In some embodiments, Figure 3AAs shown, the image is preprocessed and enhanced, and then input into the multi-task backbone network 209. The classification result of the teacher model is trained by using the newly added object to be identified, that is, the teacher target (classification i) for the newly added classification network, that is, the fully connected layer (F k ) or head network (H k )210 for adjustment.

[0123] In some embodiments, Figure 3B As shown, the trained multi-task behavior recognition model includes a fixed multi-task backbone network 209 and N second classification networks corresponding to the N tasks to be recognized, namely, a fully connected layer 211 or a head network 212.

[0124] In the embodiment of the present application, the basic network structure is that multiple tasks share a multi-task backbone network. By adding a task, a fully connected layer or head network is added without adding a new multi-task backbone network. By sharing parameters of the multi-task backbone network among various tasks, the network parameters are greatly reduced.

[0125] The present invention provides a multi-task recognition model training method, which includes:

[0126] Step S311, obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0127] Step S312: Use the sample data sets of the N tasks to be identified to train the constructed behavior recognition model to obtain a multi-task behavior recognition model, wherein the multi-task behavior recognition model is used to identify abnormal behaviors of a single person and / or multiple people; the N tasks to be identified include at least one of the following: behaviors that endanger one's own safety or violate the law, and behaviors that endanger one's own safety or violate the law of multiple people.

[0128] During the implementation process, the behavior of a single person includes climbing, falling, stealing, and damaging property; the behavior of multiple people includes climbing, fighting, falling, stealing, and damaging property.

[0129] Step S312, obtaining an image to be identified;

[0130] During the implementation process, an image to be recognized may be obtained from a video stream, which may be an online video stream obtained from a camera, that is, an online video stream shot by a camera installed in a specific environment, for example, a video stream obtained by a camera installed in an elevator or supermarket.

[0131] Step S312: input the image to be recognized into the multi-task behavior recognition model to obtain the abnormal behavior recognition result;

[0132] The abnormal behavior identification results here include the time and location of sending the abnormal behavior.

[0133] Step S312: output the abnormal behavior recognition result.

[0134] When the multi-task behavior recognition model is used to identify at least one of climbing, fighting, and falling events in the online video stream, the content corresponding to the event alarm is determined according to the event and the location and time of the event; the content corresponding to the event alarm is sent to the alarm processing platform. The corresponding terminal is determined according to the location of the event; the event alarm is sent to the terminal so that the manager holding the terminal can handle the event in time.

[0135] In the embodiment of the present application, the multi-task behavior recognition model is used to identify the abnormal behavior of a single person and multiple people; the N tasks to be identified include at least one of the following: a single person's behavior that endangers his own safety, illegal behavior; multiple people's behavior that endangers their own safety, illegal behavior. First, obtain the image to be identified in the video stream; then, input the image to be identified into the multi-task behavior recognition model to obtain the attribute information of the abnormal behavior; finally, output the attribute information of the abnormal behavior. In this way, since in the scenario of smart city management, urban safety and harmony have always been very important aspects. Therefore, when abnormal events occur, such as fighting, elevator falls, kicking and other dangerous actions, it is often necessary to be able to alarm the management personnel in the first time. The embodiment of the present application can grasp the city situation in real time. When dangerous actions occur, resulting in urban hazards, such as fighting, the system can generate an alarm in the first time, without the need for management personnel to record the video taken by the camera device 24 hours a day, which greatly saves manpower and material resources and ensures urban safety.

[0136] In the actual application of multi-task abnormal behavior recognition, the difference between videos acquired under different lighting / background / camera conditions for different tasks and videos acquired under the same conditions for the same task is very large. There are many forms of commonly used methods in the industry, such as joint-learning, learning with auxiliary tasks, and autonomous learning. It has been proven in practice that sharing parameters between different tasks can mine common features between different tasks, focusing on learning stronger and more generalized objective functions.

[0137] However, there are several important drawbacks in these methods:

[0138] Defect 1: These methods are still limited in terms of task similarity, relationship between tasks, hierarchy between tasks, and accuracy of multi-task learning in the model. Multi-task learning will cause the model to prefer categories with common features, which will affect the performance of categories with orthogonal features. Tasks that have inclusive features in the same domain and similar optimization goals can be effectively solved through multi-tasks to improve generalization. However, when the domains of different tasks are very different, multi-task parallel learning will cause the learning results to inhibit each other and the learning accuracy will be poor.

[0139] Defect 2: Multi-task abnormal behavior recognition requires a large number of task samples for training. There needs to be sufficient data support between tasks, and the number of classes needs to be balanced. Without a large number of task samples, a robust discriminator cannot be trained. Usually, methods will use category balance to alleviate this problem, but it cannot really solve the problem because the data complexity is limited and the model will tend to learn categories with higher complexity.

[0140] Defect 3. Related technologies for multi-task abnormal behavior recognition are all based on the processing of the entire video image, but the actual pre-processing has different processing modes for one person or multiple people. If trained directly, it is often easy to affect each other's accuracy.

[0141] Figure 4 A schematic diagram of a multi-task behavior recognition model training implementation process provided in an embodiment of the present application, wherein, Figure 4 As shown, the workflow is described as follows:

[0142] Step S401, obtaining N sample data sets of tasks to be identified, where N is an integer greater than or equal to 1;

[0143] The sample data sets obtained are divided into three categories: (a) Figure 1A and 1B (b) the data collected by the camera device shown; Figure 1C The crawled data of the Internet shown in Figure 1D and 1E The abnormal behavior data is generated based on the virtual game engine shown.

[0144] The above three aspects of data constitute the sample data sets for N tasks to be identified.

[0145] Step S402: using the sample data set of each of the N tasks to be identified, respectively training N first teacher models to obtain N second teacher models corresponding to the N tasks to be identified;

[0146] The tasks to be identified here can be fighting, falling, climbing, etc. The actual application data sources are relatively rich, and the cross-domain (outdoor data, indoor data) training method can be used. Through domain adaptation, the collected data, generated data, and network data learn consistent behavioral representations. The variational encoder can obtain an invariant representation of the data in each domain through different data distributions and network adjustment. The feature space between two different domains is further aligned through adversarial learning. The best model of each type of data (collected data, generated data, and network data) is obtained through domain space fusion training, and saved as a teacher model for each category to prepare for subsequent knowledge distillation.

[0147] Step S403: Acquire the first backbone network, wherein the first backbone network is a backbone network obtained by training with a preset data set;

[0148] Step S404: adjusting the number of batch normalization layers in the first backbone network to obtain a second backbone network including N batch normalization layers, wherein each batch normalization layer corresponds to a task to be identified;

[0149] Since the data distribution of each category is inconsistent, the first backbone network is adjusted to obtain a second backbone network with N batch normalization layers. Each task has its own batch normalization layer, and different batch normalization layers are used to solve the problem of inconsistent distribution of different tasks.

[0150] Here, the batch normalization layer can solve the problem of changes in the data distribution of the intermediate layers during training, so as to prevent the gradient from disappearing or exploding and speed up the training.

[0151] Since the data distribution of each category is inconsistent, the adjusted second backbone network has N batch normalization layers. Each task has its own batch normalization layer. Different batch normalization layers are used to solve the problem of inconsistent distribution of different tasks.

[0152] Step S405, obtaining feature vectors corresponding to all samples extracted in each category according to the second teacher model;

[0153] Step S406: Use the feature vectors extracted from all samples in each of the second teacher models as target features to train the second backbone network in the constructed behavior recognition model to obtain the multi-task backbone network;

[0154] The second teacher model obtained through training for each category can obtain the features of all samples in each category through each second teacher model. Through the knowledge distillation method, the features of multiple tasks are aligned with the features of a single model to obtain a better backbone network, that is, the features of a single model are used as target features to guide the multi-task behavior recognition model to train the backbone network to obtain consistent features.

[0155] Knowledge distillation: Compared with the hard output of unique hotspot, knowledge distillation can obtain more comprehensive knowledge that makes the target easier to learn, such as features or softer outputs, and separate different feature solutions through distillation. The core idea is to transfer knowledge, so as to obtain a small model that is more suitable for reasoning through a trained large model.

[0156] Step S407: Fusing the N batch normalization layers in the multi-task backbone network to obtain a third backbone network;

[0157] The parameters of the multi-task backbone network except the batch normalization layer are fixed, and N batch normalization layers are fused into one batch normalization layer. The parameters of the N batch normalization layers can be averaged first, and then the parameters are fine-tuned to obtain the third backbone network.

[0158] Step S408: training the third backbone network using the sample data sets of the N tasks to be identified, so as to adjust the parameters of the batch normalization layer in the third backbone network, and obtain a multi-task backbone network after adjustment;

[0159] like Figure 2D As shown, the image is preprocessed and enhanced, and then input into a multi-task backbone network that incorporates N batch normalization layers. The classification results of the second teacher model are used to adjust the parameters of the batch normalization layer in the third backbone network to obtain a fully adjusted multi-task backbone network.

[0160] Step S409: Use the sample data sets of the N tasks to be identified to train a first behavior recognition model to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0161] First, N first classification networks are added to the trained multi-task backbone network to obtain a first behavior recognition model. Then, the first behavior recognition model is trained using the sample data sets of the N tasks to be recognized to obtain a multi-task behavior recognition model. The behavior recognition model of the task includes the multi-task backbone network obtained by training the first backbone network and N second classification networks corresponding to the N tasks to be recognized obtained by training the N first classification networks.

[0162] like Figure 2A As shown, when N is 3, the first classification network is fully connected layer 1, fully connected layer 2, fully connected layer 3 or head network 1, head network 2, head network 3.

[0163] The multi-task behavior recognition model obtained by training is as follows Figure 3B As shown, it includes a fixed multi-task backbone network and N second classification networks corresponding one by one to the N tasks to be identified.

[0164] In the embodiment of the present application, the Internet data is crawled according to the algorithm development, and the corresponding abnormal categories are crawled for different categories, which greatly expands the abnormal data set; the corresponding abnormal categories are generated based on the virtual data of the game, and the previous related technologies do not have the ability to generate categories; the algorithm supports different tasks under multiple preprocessing in multiple scenarios, and the algorithm development solves the problem of uneven number of categories, and the previous technology does not use mining videos and generating videos to solve the problem of less data. The following beneficial effects can be achieved: 1. The collection and training of multi-source and multi-domain data is helpful for the actual verification of abnormal behavior algorithms and improves the generalization ability of behavior algorithms; 2. Through the distillation of a single model, the best accuracy of each category on each model can be maintained, and the generated model has good representation capabilities in each task; 3. Multiple tasks can be performed on a single model, which greatly reduces the video memory consumption caused by adding tasks (one task and one separate model).

[0165] Based on the foregoing embodiments, the embodiments of the present application provide a multi-task behavior recognition model training device, which includes modules, sub-modules included in each module, units included in each sub-module, and sub-units included in each unit, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0166] The present application embodiment provides a multi-task behavior recognition model device, Figure 5 Schematic diagram of the structure of the multi-task behavior recognition model training device of the present application embodiment. Figure 5 As shown, the device 500 includes:

[0167] A first acquisition module 501 is used to acquire N sample data sets of tasks to be identified, wherein the sample data sets include at least one of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1;

[0168] The first training module 502 is used to train the constructed behavior recognition model using the sample data sets of the N tasks to be identified to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, and the multi-task behavior recognition model includes a multi-task backbone network and N second classification networks, the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be identified and are obtained by training the N first classification networks.

[0169] Based on the aforementioned embodiment, the device for constructing a behavior recognition model also includes: a second acquisition module, used to obtain the first backbone network, wherein the first backbone network is a backbone network obtained by training a preset data set; and an adding model, used to add the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model.

[0170] Based on the aforementioned embodiment, the first training module includes a first training sub-module and a second training sub-module, wherein the first training sub-module is used to train the first backbone network in the constructed behavior recognition model using the sample data sets of the N tasks to be identified to obtain the multi-task backbone network; the second training sub-module is used to train the first behavior recognition model using the sample data sets of the N tasks to be identified to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

[0171] Based on the aforementioned embodiment, the first training submodule includes a first training unit and a second training unit, wherein the first training unit is used to use the sample data set of each of the N tasks to be identified to train N first teacher models respectively, and obtain N second teacher models corresponding one by one to the N tasks to be identified, wherein each of the first teacher models includes the first backbone network, a first classification network, a variational autoencoder and a discriminator; the second training unit is used to train the first backbone network using N of the second teacher models to obtain the multi-task backbone network.

[0172] Based on the foregoing embodiment, the second training unit includes an adjustment subunit, an acquisition subunit and a first training subunit, wherein the adjustment subunit is used to adjust the number of batch normalization layers in the first backbone network to obtain a second backbone network including N batch normalization layers, wherein each batch normalization layer corresponds to a task to be identified; the acquisition subunit is used to obtain feature vectors extracted from all samples in each of the N second teacher models; the training subunit is used to train the second backbone network in the constructed behavior recognition model using the feature vectors extracted from all samples in each of the second teacher models as target features to obtain the multi-task backbone network.

[0173] Based on the foregoing embodiment, the second training unit also includes a fusion subunit, a second training subunit, wherein the fusion subunit is used to fuse the N batch normalization layers in the multi-task backbone network to obtain a third backbone network; the second training subunit is used to train the third backbone network using the sample data sets of the N tasks to be identified to adjust the parameters of the batch normalization layer in the third backbone network to obtain a multi-task backbone network that has been adjusted.

[0174] Based on the foregoing embodiment, when a new task to be identified is added to the behavior recognition model of the multi-task, the device further includes a third acquisition module, a fourth acquisition module and a second training module, wherein the third acquisition module is used to acquire the behavior recognition model of the newly added task, wherein the behavior recognition model of the newly added task is obtained by adding one of the first classification networks to the behavior recognition model of the multi-task, and the added first classification network is used to classify the newly added task to be identified; the fourth acquisition module is used to acquire a new data set corresponding to the newly added task to be identified; and the second training module is used to train the behavior recognition model of the newly added task using the new data set to obtain the behavior recognition model of the newly added task that has completed the training.

[0175] Based on the above-mentioned embodiments, the multi-task behavior recognition model is used to identify abnormal behaviors of a single person and / or multiple people; the N tasks to be recognized include at least one of the following: behaviors endangering one's own safety and illegal behaviors; behaviors endangering one's own safety and illegal behaviors of multiple people. The device also includes a fifth acquisition module, an input module and an output module, wherein the fifth acquisition module is used to acquire an image to be recognized; the input module is used to input the image to be recognized into the multi-task behavior recognition model to obtain the abnormal behavior recognition result; and the output module is used to output the abnormal behavior recognition result.

[0176] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0177] It should be noted that in the embodiment of the present application, if the above-mentioned multi-task behavior model training method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a robot, a server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0178] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the multi-task behavior recognition model training method provided in the above embodiment are implemented.

[0179] Correspondingly, an embodiment of the present application provides a computer device, Figure 6 A schematic diagram of a hardware entity of an embodiment of the present application is shown in FIG. Figure 6 As shown, the hardware entity of the device 600 includes: a memory 601 and a processor 602, wherein the memory 601 stores a computer program that can be run on the processor 602, and the processor 602 implements the steps in the method provided in the above embodiment when executing the program.

[0180] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or processed by the processor 602 and various modules in the computer device 600 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0181] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0182] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0183] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0185] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0186] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0187] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0188] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a mobile phone, tablet computer, laptop computer, desktop computer, robot, server, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, magnetic disks or optical disks.

[0189] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0190] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0191] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0192] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-task behavior recognition model training method, the method comprising: Obtaining N sample data sets of tasks to be identified, wherein the sample data sets include at least two of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1; The constructed behavior recognition model is trained using the sample data sets of the N tasks to be identified to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, and the multi-task behavior recognition model includes a multi-task backbone network and N second classification networks, the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be identified and are obtained by training the N first classification networks; The behavior recognition model is obtained by the following method: Acquire the first backbone network, wherein the first backbone network is a backbone network obtained by training with a preset data set; Adding the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model; The behavior recognition model constructed by training the sample data sets of the N tasks to be recognized to obtain a multi-task behavior recognition model includes: Using the sample data sets of the N tasks to be identified, training the first backbone network in the constructed behavior recognition model to obtain the multi-task backbone network; The first behavior recognition model is trained using the sample data sets of the N tasks to be recognized to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

2. The method according to claim 1, characterized in that The method of using the sample data sets of the N tasks to be identified to train the first backbone network in the constructed behavior recognition model to obtain the multi-task backbone network includes: Using the sample data set of each of the N tasks to be identified, respectively train N first teacher models to obtain N second teacher models corresponding to the N tasks to be identified, wherein each of the first teacher models includes the first backbone network, a first classification network, a variational autoencoder and a discriminator; The first backbone network is trained using N of the second teacher models to obtain the multi-task backbone network.

3. The method according to claim 2, characterized in that The using N second teacher models to train the first backbone network to obtain the multi-task backbone network includes: Adjusting the number of batch normalization layers in the first backbone network to obtain a second backbone network including N batch normalization layers, wherein each batch normalization layer corresponds to a task to be identified; Obtain feature vectors extracted from all samples in each of the N second teacher models; The feature vectors extracted from all samples in each of the second teacher models are used as target features to train the second backbone network to obtain the multi-task backbone network.

4. The method according to claim 3, characterized in that The method further includes: fusing N batch normalization layers in the multi-task backbone network to obtain a third backbone network; The third backbone network is trained using the sample data sets of the N tasks to be identified to adjust the parameters of the batch normalization layer in the third backbone network to obtain an adjusted multi-task backbone network.

5. The method according to any one of claims 1 to 4, characterized in that: When a task to be identified is added to the multi-task behavior recognition model, the method further includes: Acquire a behavior recognition model for a newly added task, wherein the behavior recognition model for the newly added task is obtained by adding the first classification network to the behavior recognition model for the multiple tasks, and the added first classification network is used to classify the newly added task to be recognized; Acquire a new data set corresponding to the new task to be identified; The behavior recognition model of the newly added task is trained using the newly added data set to obtain a behavior recognition model of the newly added task that has completed training.

6. The method according to any one of claims 1 to 4, characterized in that: The multi-task behavior recognition model is used to identify abnormal behaviors of a single person and / or multiple people; the N tasks to be identified include at least one of the following: behaviors that endanger the personal safety of a single person or illegal behaviors; behaviors that endanger the personal safety of multiple people or illegal behaviors.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain an image to be recognized; Inputting the image to be recognized into the multi-task behavior recognition model to obtain an abnormal behavior recognition result; The abnormal behavior recognition result is outputted.

8. A multi-task behavior recognition model training device, the device comprising: A first acquisition module is used to acquire N sample data sets of tasks to be identified, wherein the sample data sets include at least two of virtual data, data collected by a camera device, and crawled data from the Internet, and N is an integer greater than or equal to 1; A first training module is used to train the constructed behavior recognition model using the sample data sets of the N tasks to be identified, so as to obtain a multi-task behavior recognition model, wherein the constructed behavior recognition model includes a first backbone network and N first classification networks, and the multi-task behavior recognition model includes a multi-task backbone network and N second classification networks, wherein the multi-task backbone network is obtained by training the first backbone network, and the N second classification networks correspond one-to-one to the N tasks to be identified, and are obtained by training the N first classification networks; The device for constructing the behavior recognition model further includes: A second acquisition module is used to acquire the first backbone network, wherein the first backbone network is a backbone network obtained by training with a preset data set; Adding a model, used to add the N first classification networks on the basis of the first backbone network to obtain the constructed behavior recognition model; The first training module includes a first training submodule and a second training submodule. The first training submodule is used to train the first backbone network in the constructed behavior recognition model using the sample data sets of the N tasks to be recognized, so as to obtain the multi-task backbone network; The second training submodule is used to train a first behavior recognition model using the sample data sets of the N tasks to be recognized to obtain the multi-task behavior recognition model, wherein the first behavior recognition model includes the multi-task backbone network and the N first classification networks.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the computer program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A storage medium, characterized in that: Computer instructions are stored, and when the computer instructions are executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images