Training method, device, equipment, medium and program product for behavior recognition model

Abstract

The embodiment of the present application discloses a method for training a behavior recognition model, including: obtaining a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image; using the training sample set to train a behavior recognition model built based on a generative adversarial network. The embodiment of the present application also provides a training device, equipment, storage medium and program product for a behavior recognition model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and is related to, but not limited to, training methods and devices, equipment, media, and program products for behavior recognition models. Background Art

[0002] For video understanding tasks, different data reading formats and inefficient pre-processing greatly restrict the efficient training of models driven by massive data. In addition, unlike low-level visual tasks such as denoising and super-resolution, high-level recognition and classification tasks are very sensitive to different decoding and pre-processing schemes. When training and deployment use different pre-processing logic, it will greatly affect the deployment accuracy. However, because different hardware platforms often provide different decoding and pre-processing implementations, there is an inevitable misalignment between offline testing and online deployment. Summary of the invention

[0003] The embodiments of the present application provide a method and apparatus, device, medium and program product for training a behavior recognition model.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for training a behavior recognition model, the method comprising:

[0006] Acquire a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image;

[0007] The training sample set is used to train a behavior recognition model built based on a generative adversarial network.

[0008] In some possible implementations, obtaining the training sample set includes: randomly selecting at least one decoding method and / or at least one data enhancement scheme from a preprocessing library to preprocess each of the sample images to obtain the training sample set; the preprocessing library is also used to include at least two of the decoding methods and at least two of the data enhancement schemes.

[0009] In this way, by setting the preprocessing library, a variety of methods are used to perform hybrid decoding and data enhancement processing on different sample images during the training process, thereby improving the compatibility and robustness of decoding / preprocessing differences on different hardware platforms.

[0010] In some possible implementations, the randomly selecting at least one decoding method and at least one data enhancement scheme from a preprocessing library to preprocess each of the sample images to obtain the training sample set includes: randomly selecting one of the decoding methods from the preprocessing library to perform data decoding on the at least one sample image; randomly selecting one of the data enhancement schemes to perform data enhancement processing on each of the decoded sample images to obtain the training sample set.

[0011] In this way, in view of the differences in decoding and preprocessing implementation schemes on different hardware platforms, by adopting multiple mixed decoding / data enhancement processing methods during the training process, the compatibility and robustness of the model to the decoding / preprocessing differences on different hardware platforms when deployed online are improved, and the online and offline testing processes and accuracy are aligned.

[0012] In some possible implementations, the data enhancement scheme includes a geometric transformation operation and a color transformation operation, and the randomly selecting one of the data enhancement schemes to perform data enhancement processing on each of the decoded sample images to obtain the training sample set includes: performing a random geometric transformation operation and / or a color transformation operation on each of the decoded sample images to obtain the training sample set.

[0013] In this way, when the sample images obtained are limited, the diversity of training samples can be increased through geometric transformation operations or color transformation operations. While improving the efficient training of the model, the robustness of the model for pre-deployment processing is enhanced to avoid overfitting.

[0014] In some possible implementations, the method further includes: using a unified data loading interface to perform format conversion on images in different storage formats to obtain the at least one sample image in a unified format.

[0015] In this way, images in different storage formats are unified to obtain image data in the same format, thereby facilitating subsequent decoding and data enhancement processing of the image data.

[0016] In some possible implementations, the training sample set includes real annotated data and generated virtual sample data; the constructed behavior recognition model includes a backbone network, a classifier, a generator and a discriminator respectively connected to the output of the backbone network; using the training sample set to train the behavior recognition model constructed based on the generative adversarial network includes: alternately fixing the network parameters of the first part and the second part of the behavior recognition model at a specific frequency; wherein the first part includes the backbone network, the classifier and the generator, and the second part includes the discriminator; while fixing the network parameters of one part of the behavior recognition model, using the real annotated data and the virtual sample data to iteratively train the remaining part of the behavior recognition model; wherein the one part is the first part or the second part.

[0017] In this way, by using a large amount of generated virtual sample data to assist in the training of the business real scene behavior recognition model, the problem of data scarcity is effectively alleviated. At the same time, virtual sample data is used to assist a small amount of real annotated data of a specific category to perform adversarial training of a specific frequency, achieving cross-domain learning in the feature space and improving the overall performance of the model for abnormal event detection.

[0018] In some possible implementations, while fixing the network parameters of a part of the behavior recognition model, the remaining part of the behavior recognition model is iteratively trained using real labeled data and generated virtual sample data, including: while fixing the network parameters of the first part, performing feature extraction on the real labeled data and the virtual sample data through the backbone network to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data; and iteratively training the discriminator using the first feature vector and the second feature vector.

[0019] In this way, the trained backbone network extracts features from the input real labeled data and virtual sample data to obtain the feature vectors of the real labeled data and virtual sample data, and then uses the feature vectors to iteratively train the discriminator to distinguish between real and fake sample data. After multiple cycles of iterative training, the discriminator can accurately determine whether the feature vector output by the backbone network belongs to real labeled data or virtual sample data.

[0020] In some possible implementations, while fixing the network parameters of a part of the behavior recognition model, the remaining part of the behavior recognition model is iteratively trained using real labeled data and generated virtual sample data, including: while fixing the network parameters of the second part, labeling the feature vector output by the backbone network with true or false labels by the discriminator; iteratively training the classifier using the data corresponding to the feature vector carrying the true label and the data corresponding to the feature vector carrying the false label; iteratively training the generator using the data corresponding to the feature vector carrying the false label.

[0021] In this way, the feature vector output by the backbone network is labeled with true or false labels by the trained discriminator, and then the labeled real data and virtual data are used to reversely train the classifier to predict the feature vector output by the backbone network to obtain the prediction score of abnormal behavior; and the labeled virtual data is used to reversely train the generator to determine the ability of the true label for the virtual sample data. In this way, after several cycles of iterative training, the classifier obtained can accurately predict abnormal behavior in video data, and the generator can generate virtual data that is as real as possible.

[0022] In some possible implementations, while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using real labeled data and generated virtual sample data includes: determining the classification loss of the classifier and the adversarial loss of the discriminator; taking a weighted sum of the classification loss and the adversarial loss to determine the optimization objective function of the behavior recognition model; and according to the optimization objective function, while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using the real labeled data and the virtual sample data.

[0023] In this way, the optimization objective function of the behavior recognition model is determined by weighted summation of the classification loss and adversarial loss. Thus, by optimizing the objective function, the discriminator loss is maximized and the classification loss is minimized, thus improving the stability and accuracy of model training.

[0024] In some possible implementations, the method further includes: obtaining an updated optimization objective function by adjusting the weight of the adversarial loss; and iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing one part of the network parameters of the behavior recognition model according to the optimization objective function, including: iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing one part of the network parameters of the behavior recognition model according to the updated optimization objective function.

[0025] In this way, by adjusting the weight of the adversarial loss through this design to achieve the optimization goal of maximizing the discriminator loss and minimizing the classification loss, better performance can be obtained than direct hybrid training.

[0026] In some possible implementations, while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using real labeled data and generated virtual sample data includes: determining the classification loss of the virtual sample data passing through the classifier; iteratively filtering the first sample data whose classification loss is greater than or equal to a first threshold; wherein the value of the first threshold is sequentially increased to a fixed value in each filtering process; while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using the real labeled data and the filtered virtual sample data.

[0027] In this way, when the training reaches a certain stage, the virtual sample data with large classification loss, i.e. low quality, is gradually filtered out according to the prediction score of the classifier, and then the high-quality virtual sample data and a small amount of real collected data are retained for adversarial training, thereby eliminating the noise data in the sample and reducing unnecessary model fitting. As the model training progresses, only highly similar valid virtual sample data is retained for auxiliary adversarial training, which improves the overall performance of the model for abnormal event detection and alleviates the dilemma of scarce abnormal data and high noise in generated data.

[0028] In some possible implementations, while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using the real labeled data and the filtered virtual sample data includes: using the filtered virtual sample data as second sample data; when the classification loss of the second sample data is greater than or equal to a second threshold, increasing the weight of the second sample data; wherein the second threshold is less than the first threshold; while fixing a portion of the network parameters of the behavior recognition model, iteratively training the remaining portion of the behavior recognition model using the real labeled data and the second sample data.

[0029] In this way, for the second sample data that is more difficult to identify than other samples and results in greater classification loss, the model's learning ability for difficult samples is enhanced by increasing the weight of the second sample data during the training process. Under the premise of avoiding misfitting of noise sample data, online dynamic difficult sample mining is achieved.

[0030] In some possible implementations, when the classification loss of the second sample data is greater than or equal to a second threshold, increasing the weight of the second sample data includes: determining, based on the classification loss of the second sample data, indistinguishable sample data of an intermediate difficulty level as new second sample data; and increasing the weight of the new second sample data when the classification loss of the new second sample data is greater than the second threshold.

[0031] In this way, in order to avoid the impact of increased noise data due to blindly determining the weight of the second sample data based on the size of the classification loss value, only online mining of medium-difficulty difficult samples can be performed to prevent misfitting of noise data while improving the recall of truly difficult samples.

[0032] In some possible implementations, the method further includes: acquiring original video stream data; preprocessing the original video stream data to obtain a video sequence to be identified; identifying the video sequence to be identified using the trained behavior recognition model to determine abnormal behavior in the video sequence to be identified.

[0033] In this way, since the trained behavior recognition model is more robust to deployment on different hardware platforms, when the behavior recognition model is used to identify offline video sequences to be identified, the accuracy aligned with that during online deployment can be obtained.

[0034] In a second aspect, an embodiment of the present application provides a training device for a behavior recognition model, including a sample acquisition module and a model training module, wherein:

[0035] The sample acquisition module is used to acquire a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image;

[0036] The model training module is used to use the training sample set to train a behavior recognition model built based on a generative adversarial network.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the steps in the training method of the above-mentioned behavior recognition model are implemented.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the training method of the above-mentioned behavior recognition model.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program, comprising a computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes the steps in the training method configured to implement the above-mentioned behavior recognition model.

[0040] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the steps in the training method of the above-mentioned behavior recognition model.

[0041] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0042] In an embodiment of the present application, first, at least one sample image obtained after data decoding and data enhancement processing by any data preprocessing scheme is obtained to form a training sample set; then, the training sample set is used to train a behavior recognition model built based on a generative adversarial network, which can target different decoding and preprocessing implementation schemes for different hardware platforms. By adopting a variety of mixed decoding / preprocessing methods during the training process, the efficient training of the model is improved while enhancing the robustness of the behavior recognition model for pre-processing before deployment on different hardware platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0044] Figure 1 A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0046] Figure 3A A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0047] Figure 3B A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0048] Figure 3C A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0049] Figure 4AA flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0050] Figure 4B A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0051] Figure 4C A flowchart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0052] Figure 5A A logical flow chart of a method for training a behavior recognition model provided in an embodiment of the present application;

[0053] Figure 5B A system block diagram of a method for training a behavior recognition model provided in an embodiment of the present application;

[0054] Figure 5C A schematic diagram of a unified data loading interface provided in an embodiment of the present application;

[0055] Figure 5D A schematic diagram of a robust preprocessing solution provided in an embodiment of the present application;

[0056] Figure 5E A schematic diagram of automatic data enhancement processing provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of a training device for a behavior recognition model provided in an embodiment of the present application;

[0058] Figure 7 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The following embodiments are used to illustrate the present application, but are not intended to limit the scope of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0060] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0061] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0062] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those skilled in the art in the field to which the embodiments of the present application belong. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0063] The solution provided in the embodiments of the present application involves artificial intelligence technology, which is specifically described by the following embodiments:

[0064] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0065] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The machine learning technology involved in the embodiments of this application.

[0066] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0067] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0068] Backbone network: refers to the network that extracts features. Its function is to extract information from the image for use by the subsequent network.

[0069] Fully Connected Layer (FC): It plays the role of a "classifier" in the entire convolutional neural network, that is, it classifies the features extracted by the previous layer.

[0070] Binary classification: Indicates that there are two categories in the classification task. For example, when classifying the source of data, you may get two different results: the source of the data is real data or virtual data. Binary classification assumes that each sample is set with one and only one label. A data can be real data or virtual data, but not both at the same time.

[0071] Loss function: It is used to evaluate the difference between the model's predicted value and the true value. The better the loss function, the better the model's performance. Different models generally use different loss functions.

[0072] Cross-entropy loss function: It is essentially a log-likelihood function that can be used in binary and multi-classification tasks.

[0073] Generative Adversarial Networks (GAN) is a powerful deep learning algorithm for sample generation. Generative adversarial networks include generative networks and discriminative networks. Traditional generative adversarial networks first train the generator based on random noise generated samples and real samples, and use real samples and samples generated by the generator to train the discriminator so that it can distinguish between real samples and generated pseudo samples as much as possible. Then the generative network is trained to minimize the gap between training samples and generated samples. The two are iteratively updated in a cycle, and finally the distribution of generated samples and real samples is consistent, thereby generating high-quality difficult samples. The samples generated by generative adversarial networks are more diverse, but due to the large randomness, the quality of the generated samples is not very stable.

[0074] In the actual training process of the generative adversarial network, there are two training processes: one is that the discriminator uses real data and fake data to train at the same time; the other is to input fake data (but the label is true) into the combined generated generative adversarial network, and use this part of the error to update the generator. In other words, the discriminator will improve its ability to identify real data and fake data as much as possible, while the generator will generate real data as much as possible; the discriminator uses real data and fake data together to train to update the weights, while the generator can only use the complete model to feedback the discriminator's error to the front to achieve weight update.

[0075] The embodiment of the present application provides a method for training a behavior recognition model, which is applied to electronic devices. The electronic devices include but are not limited to mobile phones, laptops, tablet computers and handheld Internet devices, multimedia devices, streaming media devices, mobile Internet devices, wearable devices or other types of devices. The functions implemented by the method can be implemented by calling program codes by a processor in the electronic device. Of course, the program codes can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium. The processor can be used to process the training process of the behavior recognition model, and the memory can be used to store the data required and the data generated during the training process of the behavior recognition model.

[0076] Figure 1 A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises at least the following steps:

[0077] Step S110, obtaining a training sample set.

[0078] Here, the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image. The sample image is a description of the detailed information such as shape, texture, grayscale, etc. of the image content in the video or image sequence to be recognized. For the behavior recognition model, the sample image can be the real annotated data obtained by performing abnormal behavior detection on the collected video source data, or it can be the virtual sample data generated by the game engine.

[0079] Data preprocessing is very important when building a network model and can often determine the training results. Of course, for different data sets, the preprocessing methods will have more or less particularity and limitations. Unlike low-level visual tasks such as denoising and super-resolution, high-level recognition and classification tasks are very sensitive to different decoding methods and data enhancement processing schemes. When training and deployment use different pre-processing logics, the deployment accuracy will be greatly affected. During the training process, the embodiment of the present application uses a variety of methods to perform hybrid decoding and data enhancement processing on different sample images, so that the model is insensitive to differences in data decoding and data enhancement processing on different hardware platforms, which significantly improves the robustness of the model to pre-deployment processing and aligns the online and offline test processes and accuracy.

[0080] In implementation, a preprocessing library including preprocessing methods for all hardware platforms can be pre-set, and then a decoding method is randomly selected from the preprocessing library to process the sample image during the training process, so that the model is insensitive to the differences between different encoders, or a data enhancement scheme is randomly selected to perform data enhancement on the sample, so that the model is compatible with the hybrid data enhancement implementation scheme, and a mixture of multiple encoding methods and data enhancement processing schemes can be used at the same time, thereby improving the efficient training of the model while enhancing the robustness of the model to pre-deployment processing.

[0081] Step S120, using the training sample set to train a behavior recognition model built based on a generative adversarial network.

[0082] Here, the behavior recognition model built based on the generative adversarial network can be obtained by connecting the classifier, the generator and the discriminator respectively after the output of the backbone network.

[0083] The backbone network is used to extract features from real labeled data and virtual sample data, the classifier is used to determine the prediction score of abnormal behavior for the feature vector output by the backbone network, the discriminator is used to generate a discriminant score for the feature vector output by the backbone network, and the generator is used to generate an evaluation score for the virtual sample data. The discriminator tries to distinguish whether the feature vector belongs to real labeled data or virtual sample data, while the evaluation score generated by the generator tries to label the virtual sample data with the real label, thereby realizing adversarial training of real sample data and virtual sample data.

[0084] In an embodiment of the present application, first, at least one sample image obtained after data decoding and data enhancement processing by any data preprocessing scheme is obtained to form a training sample set; then, the training sample set is used to train a behavior recognition model built based on a generative adversarial network, which can target different decoding and preprocessing implementation schemes for different hardware platforms. By adopting a variety of mixed decoding / preprocessing methods during the training process, the efficient training of the model is improved while enhancing the robustness of the behavior recognition model for pre-processing before deployment on different hardware platforms.

[0085] In some possible embodiments, the training sample set can be obtained in the following manner: randomly selecting at least one decoding method and / or at least one data enhancement scheme from a preprocessing library to preprocess each sample image to obtain the training sample set; the preprocessing library includes at least two of the decoding methods and at least two of the data enhancement schemes. In this way, by setting a preprocessing library including multiple decoding methods and multiple data enhancement schemes, multiple methods are used to perform mixed decoding and data enhancement processing on different sample images during the training process, thereby improving the compatibility and robustness of decoding / preprocessing differences on different hardware platforms.

[0086] Figure 2 A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the above step S110 "obtaining a training sample set" can be implemented by the following steps:

[0087] Step S210, using a unified data loading interface, converting images in different storage formats to obtain at least one sample image in a unified format;

[0088] Here, the unified data loading interface may be a data conversion function implemented by a middleware between the application system and the system software, which can unify data in different formats into the same interface mode.

[0089] It is understandable that the original images stored in different hardware platforms such as parallel distributed file systems (Lustre), distributed file systems (Ceph), and object systems (Object Systems Group, OSG) have different data reading formats. If the original images obtained from these different middlewares are directly used as sample images, it will greatly restrict the efficient training of models driven by massive data. Therefore, the embodiments of the present application perform unified format conversion on images of different storage formats, such as adding self-description and annotation information to the original images to form a retrievable AI data format, making training data processing more efficient.

[0090] Step S220, randomly selecting a decoding method from the preprocessing library to perform data decoding on at least one sample image;

[0091] Here, by deploying multiple autoencoders online and randomly selecting an autoencoder to decode the sample image to be recognized, the behavior recognition model can be made insensitive to the differences in different data decoding methods, thereby improving the robustness of the preprocessing deployment.

[0092] It should be noted that the autoencoder is an unsupervised neural network model that can learn the implicit features of the input data, which is called coding. Using the autoencoder to "compress" and reduce the dimension of the data can not only "reduce the pressure" of the machine, but also facilitate data visualization. At the same time, the original input data can be reconstructed using the learned new features, which is called decoding. By using a variety of mixed decoding methods during the training process, the robustness of the behavior recognition model for pre-deployment processing is enhanced while improving the efficient training of the model.

[0093] Step S230: randomly select a data enhancement scheme from the preprocessing library to perform data enhancement processing on at least one decoded sample image to obtain a training sample set.

[0094] Here, data enhancement is also called data amplification, which means making limited data produce the equivalent value of more data without substantially increasing the data. The essence of data enhancement is to enhance the generalization ability of the model. Data enhancement can be divided into supervised data enhancement and unsupervised data enhancement methods. Common data enhancement implementation schemes include geometric transformations such as cropping, flipping, rotation, scaling, and distortion, as well as pixel perturbation, adding noise, lighting adjustment, contrast adjustment, sample summation or interpolation, segmentation patches, etc. In addition, there are some data enhancements using reinforcement learning.

[0095] In a possible implementation, the data enhancement process includes a geometric transformation operation and a color transformation operation, and a random geometric transformation operation and / or color transformation operation is performed on each of the decoded sample images to obtain the training sample set.

[0096] Here, the geometric transformation category refers to the geometric transformation of the image, including flipping, rotating, cropping, deforming, scaling and other operations, without changing the content of the image itself. This method may select a part of the image or redistribute the pixels. If the content of the image itself is to be changed, it belongs to the color transformation category of data enhancement, which commonly includes noise, blur, color change, erasure, filling, etc. Therefore, when the sample images obtained are limited, the diversity of training samples can be increased through geometric transformation operations or color transformation operations, which can improve the efficient training of the model while avoiding overfitting.

[0097] In the embodiment of the present application, the images of different storage formats are firstly unified to obtain image data of the same format, and then a unified data loading interface is provided in the model training stage, and random decoding and preprocessing and other data enhancement schemes are selected for the video data. In this way, the compatibility and robustness of the model with decoding / preprocessing differences on different hardware platforms are improved when it is deployed online, and the problem of inconsistent / lack of robustness of online and offline platform deployment is solved.

[0098] In some possible embodiments, the training sample set includes real labeled data and generated virtual sample data; the constructed behavior recognition model includes a backbone network, a classifier, a generator and a discriminator respectively connected to the output of the backbone network. Figure 3A A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 3A As shown, the above step S120 "using the training sample set to train the behavior recognition model built based on the generative adversarial network" can be implemented by the following steps:

[0099] Step S310, alternately fixing the network parameters of the first part and the second part of the behavior recognition model at a specific frequency;

[0100] Here, the specific frequency may be a preset switching frequency, such as a switching frequency f=2, 5, 10... times / second. In some embodiments, the frequency of alternating training may be determined based on a fixed number of iterations; in other embodiments, the frequency of alternating training may also be determined based on a change in loss.

[0101] The first part includes the backbone network, the classifier and the generator, and the second part includes the discriminator. The network parameters of the first part and the second part of the behavior recognition model are alternately fixed, that is, the first part and the second part are trained separately.

[0102] Step S320, while fixing the network parameters of a part of the behavior recognition model, iteratively train the remaining part of the behavior recognition model using the real labeled data and the virtual sample data.

[0103] Here, the real annotated data is annotated by annotating the real data of a specific category in the video or image sequence to be identified. For abnormal behavior detection in road traffic scenarios, the real annotated data is the lying behavior annotated data, the falling behavior annotated data, etc. Usually, these categories of abnormal behaviors are low-probability events, making this part of the real annotated data relatively scarce, so it is necessary to generate a large amount of virtual sample data to assist the real annotated data in cold starting the model.

[0104] Here, the virtual sample data is generated by the game engine, and in implementation, any data can be generated by setting different parameters (including viewing angle, weather, lighting, scene, etc.).

[0105] The embodiment of the present application uses the data generated by the game engine as virtual sample data to assist in the training of the business real-scenario behavior recognition model, which effectively alleviates the problem of data scarcity and improves the overall performance of the model for abnormal event detection.

[0106] Here, the one part is the first part or the second part. The network parameters of one part of the above-mentioned behavior recognition model are fixed to iteratively train the remaining part, that is, first fix the network parameters of the backbone network, classifier and generator to train the discriminator to obtain the trained discriminator; then fix the network parameters of the trained discriminator, and perform back propagation training on the classifier and generator.

[0107] For example, the specific frequency, such as the switching frequency of iterative training, is set to twice per second. Within 0 to 0.5 seconds, the network parameters of the backbone network, classifier, and generator in the behavior recognition model are fixed, and the discriminator is trained using real labeled data and virtual sample data to complete the first round of training; within 0.5 to 1 second, the network parameters of the discriminator in the previous round of training are fixed, and the backbone network, classifier, and generator are trained using real labeled data and virtual sample data to complete the second round of training; within 1 second to 1.5 seconds, the network parameters of the backbone network and other network parameters of the second round of training are fixed, and the discriminator is continued to be trained to complete the third round of training; in this way, a part of the behavior recognition model is fixed every 0.5 seconds and the remaining part is iteratively trained until the network parameters of each part meet the requirements to obtain a trained behavior recognition model.

[0108] In some possible implementations, the process of training the discriminator can be implemented through the following steps: while fixing the network parameters of the first part, performing feature extraction on the real labeled data and the virtual sample data through the backbone network to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data; and iteratively training the discriminator using the first feature vector and the second feature vector.

[0109] The above process uses the trained backbone network to extract features from the input real labeled data and virtual sample data, obtains the feature vectors of the real labeled data and virtual sample data, and then uses the feature vectors to iteratively train the discriminator to distinguish between real and fake sample data. After multiple cycles of iterative training, the discriminator can finally make the score of the feature vector output of the real labeled data as close to 1 as possible, and the score of the feature vector output of the virtual sample data as close to 0 as possible, that is, the discriminator can accurately determine whether the feature vector output by the backbone network belongs to the real labeled data or the virtual sample data.

[0110] In some possible implementations, the process of training the classifier and the generator can be implemented by the following steps: while fixing the network parameters of the second part, labeling the feature vector output by the backbone network with a true or false label by the discriminator; iteratively training the classifier using the data corresponding to the feature vector carrying the true label and the data corresponding to the feature vector carrying the false label; iteratively training the generator using the data corresponding to the feature vector carrying the false label.

[0111] The above process uses the trained discriminator to label the feature vector output by the backbone network with true or false labels, and then uses the labeled real data and virtual data to reversely train the classifier to predict the feature vector output by the backbone network to obtain the prediction score of abnormal behavior; and uses the labeled virtual data to reversely train the generator to determine the ability of the true label for the virtual sample data. In this way, the classifier obtained after several cycles of iterative training can accurately predict abnormal behavior in video data, and the generator obtained can make the score of the feature vector output of the virtual sample data as close to 1 as possible, that is, to label the virtual sample data with the true label as much as possible, thereby forming a mutual confrontation with the discriminator, and finally achieving an ideal equilibrium state and a generation effect that is close to the real labeled data.

[0112] In some possible implementations, the optimization target can be determined by the following process to complete the above-mentioned iterative training: determine the classification loss of the classifier and the adversarial loss of the discriminator; perform weighted summation of the classification loss and the adversarial loss to determine the optimization objective function of the behavior recognition model; according to the optimization objective function, while fixing the network parameters of a part of the behavior recognition model, use the real labeled data and the virtual sample data to iteratively train the remaining part of the behavior recognition model.

[0113] The above process determines the optimization objective function of the behavior recognition model by weighted summing the classification loss and adversarial loss. The optimization objective function is used to continuously update the optimization network parameters to maximize the discriminator loss and minimize the classification loss, thereby improving the stability and accuracy of model training.

[0114] In some possible implementations, the weight of the adversarial loss can be adjusted to obtain an updated optimization objective function; according to the updated optimization objective function, the remaining part of the behavior recognition model is iteratively trained using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model. By adjusting the weight of the adversarial loss in this design to achieve the optimization goal of maximizing the discriminator loss and minimizing the classification loss, better performance can be obtained than direct hybrid training.

[0115] In an embodiment of the present application, virtual sample data is used to assist a small amount of real annotated data of a specific category in adversarial training of a specific frequency, so as to achieve cross-domain learning in the feature space and improve the overall performance of the model for abnormal event detection. During the training process, the network parameters of the backbone network, classifier, and generator are first fixed to train the discriminator to obtain a trained discriminator; then the network parameters of the trained discriminator are fixed, and the classifier and generator are back-propagated. The training is iterated in this way, so that the generator provides virtual sample data for the discriminator for training, and the discriminator provides optimization targets for the generator and classifier, and finally the optimal behavior recognition model is obtained.

[0116] Figure 3B A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 3B As shown, the above step S320 "iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model" can be implemented by the following process:

[0117] Step S3201, determining the classification loss of the classifier and the adversarial loss of the discriminator;

[0118] Here, the feature vectors of the real labeled data and the feature vectors of the virtual sample data extracted by the backbone network are used as the input of the classifier, so that the classifier outputs the prediction score of abnormal behavior, and classification loss is generated in the process; at the same time, the feature vector of the virtual sample data labeled by the generator extracted by the backbone network is used as the input of the discriminator, so that the discriminator distinguishes the true and false data corresponding to the feature vector, and generates adversarial loss.

[0119] Step S3202, performing weighted summation of the classification loss and the adversarial loss to determine the optimization objective function of the behavior recognition model;

[0120] Here, the optimization objective function of the behavior recognition model is determined by weighted summation of the classification loss and adversarial loss. The optimization objective function is used to continuously update the optimization network parameters to maximize the discriminator loss and minimize the classification loss, thereby improving the stability and accuracy of model training.

[0121] Step S3203, when the network parameters of the first part are fixed, feature extraction is performed on the real labeled data and the virtual sample data through the backbone network to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data;

[0122] Step S3204, iteratively training the discriminator using the first eigenvector and the second eigenvector according to the optimization objective function.

[0123] Here, the trained discriminator can make the score of the feature vector output of the real labeled data as close to 1 as possible, and the score of the feature vector output of the virtual sample data as close to 0 as possible. In other words, the discriminator can accurately determine whether the feature vector output by the backbone network belongs to the real labeled data or the virtual sample data.

[0124] In the embodiment of the present application, the optimization objective function of the behavior recognition model is determined by weighted summation of the two loss functions, classification loss and adversarial loss. Thus, by optimizing the objective function, the discriminator loss is maximized and the classification loss is minimized, thereby improving the stability and accuracy of the model training. After multiple cycles of iterative training, the discriminator can accurately determine whether the feature vector output by the backbone network belongs to real labeled data or virtual sample data.

[0125] Figure 3C A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 3C As shown, the above step S320 "iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model" can be implemented by the following process:

[0126] Step S3211, determining the classification loss of the classifier and the adversarial loss of the discriminator;

[0127] Step S3212, performing weighted summation on the classification loss and the adversarial loss to determine an optimization objective function of the behavior recognition model;

[0128] Step S3213, while fixing the network parameters of the second part, labeling the feature vector output by the backbone network with a true or false label through a discriminator;

[0129] Here, the trained discriminator can accurately determine whether the feature vector output by the backbone network belongs to real labeled data or virtual sample data, thereby labeling the feature vector output by the backbone network with a true or false label.

[0130] Step S3214, iteratively training the classifier using the data corresponding to the feature vector carrying the true label and the data corresponding to the feature vector carrying the false label according to the optimization objective function;

[0131] Here, the labeled real data and virtual data are used to reversely train the classifier to predict the feature vector output by the backbone network to obtain the prediction score of abnormal behavior.

[0132] Step S3215, according to the optimization objective function, the generator is iteratively trained using the data corresponding to the feature vector carrying the false label.

[0133] Here, the generator is trained backwards using labeled virtual data to determine the ability of the true label for the virtual sample data. In this way, after several cycles of iterative training, the classifier obtained can accurately predict abnormal behaviors in video data, and the generator obtained can make the score of the feature vector output of the virtual sample data as close to 1 as possible.

[0134] In the embodiment of the present application, the optimization objective function of the behavior recognition model is determined by weighted summation of the two loss functions, classification loss and adversarial loss. Thus, by optimizing the objective function, the discriminator loss is maximized and the classification loss is minimized, thereby improving the stability and accuracy of model training. In this way, the classifier obtained after several cycles of iterative training can accurately predict abnormal behaviors in video data, and the generator can label virtual sample data with real labels as much as possible, thereby forming a mutual confrontation with the discriminator, and finally achieving an ideal equilibrium state and a generation effect that is close to the real labeled data.

[0135] Figure 4A A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 4A As shown, the above step S320 "iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model" can be implemented by the following process:

[0136] Step S410a, determining the classification loss of the virtual sample data through the classifier;

[0137] Here, the classifier predicts the features of the virtual sample data extracted by the backbone network to obtain a predicted score for abnormal behavior. If the predicted score is higher than a certain threshold, it is judged that abnormal behavior exists in the virtual sample data. In the case where the loss function is a cross entropy loss, the cross entropy of the predicted score and the label can be further calculated to determine the classification loss. As for which loss function is used in actual implementation, the embodiment of the present application does not limit this.

[0138] Step S420a, iteratively filtering the first sample data whose classification loss is greater than or equal to the first threshold;

[0139] Here, the first threshold represents the maximum loss value of sample data allowed in model training, and the value of the first threshold is increased to a fixed value in each filtering process. In other words, the first sample data, i.e., low-quality sample data with large classification loss, is gradually filtered according to the prediction score. Therefore, as the model training proceeds, only highly similar valid virtual sample data is retained for auxiliary adversarial training, which improves the overall performance of the model for abnormal event detection and alleviates the dilemma of scarce abnormal data and high noise in generated data.

[0140] Step S430a, using the filtered virtual sample data as second sample data;

[0141] Step S440a, when the classification loss of the second sample data is greater than or equal to the second threshold, increase the weight of the second sample data;

[0142] Here, the second threshold is smaller than the first threshold; the first sample data with too large classification loss has been filtered out by the first threshold, and the remaining second sample data is judged whether its classification loss is greater than the second threshold.

[0143] For the second sample data that is more difficult to identify than other samples and results in greater classification loss, the model's learning ability for difficult samples is enhanced by increasing the weight of the second sample data during training. Under the premise of avoiding misfitting of noise sample data, online dynamic difficult sample mining is achieved.

[0144] In some possible embodiments, before performing difficult sample mining, the second sample data may also be divided into sample difficulty levels. For example, based on the classification loss of the second sample data, difficult sample data of an intermediate difficulty level may be determined from the second sample data as new second sample data. When the classification loss of the new second sample data is greater than a second threshold, the weight of the new second sample data may be increased.

[0145] In order to avoid the impact of increased noise data caused by blindly determining the weight of the second sample data based on the classification loss value, the above process can only perform online mining of medium-difficulty difficult samples, preventing misfitting of noise data while improving the recall of truly difficult samples.

[0146] Step S450a, while fixing the network parameters of a part of the behavior recognition model, iteratively train the remaining part of the behavior recognition model using the real labeled data and the second sample data.

[0147] Here, the retained second sample data is used to assist the real labeled data for adversarial training.

[0148] In the embodiment of the present application, when training reaches a certain stage, virtual sample data with large classification loss, i.e., low quality, is gradually filtered out according to the prediction score of the classifier, and then high-quality virtual sample data and a small amount of real collected data are retained for adversarial training, thereby eliminating noise data in the sample and reducing unnecessary model fitting. At the same time, under the premise of avoiding misfitting of noise sample data, online dynamic difficult sample mining is achieved.

[0149] Figure 4B A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 4B As shown, the above step S320 "iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model" can be implemented by the following process:

[0150] Step S410b, determining the classification loss of the virtual sample data through the classifier;

[0151] Step S420b, iteratively filtering the first sample data whose classification loss is greater than or equal to the first threshold;

[0152] Here, the value of the first threshold in each filtering process is successively increased to a fixed value, thereby gradually eliminating the noise sample data in the virtual sample data, and only highly similar valid virtual sample data is retained for auxiliary adversarial training.

[0153] Step S430b, while fixing the network parameters of the second part, labeling the feature vector output by the backbone network with a true or false label by a discriminator;

[0154] Step S440b, iteratively training the classifier using the data corresponding to the feature vector carrying the true label and the data corresponding to the feature vector carrying the false label;

[0155] Step S450b, iteratively train the generator using the data corresponding to the feature vector carrying the false label.

[0156] In the embodiment of the present application, in order to reduce the interference of data noise, the model is used to dynamically filter low-quality generated samples and retain high-quality video samples during the training process, and then adversarial training is performed on the continuously retained virtual sample data and a small amount of real labeled data. The feature vector output by the backbone network is labeled with true or false labels by the trained discriminator, and then the labeled real data and virtual data are used to reversely train the classifier and generator. In this way, the classifier obtained after several cycles of iterative training can accurately predict abnormal behavior in video data, and the generator can generate virtual data that is as real as possible.

[0157] Figure 4C A flow chart of a method for training a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 4C As shown, the above step S320 "iteratively training the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of a part of the behavior recognition model" can be implemented by the following process:

[0158] Step S410c, determining the classification loss of the virtual sample data through the classifier;

[0159] Step S420c, iteratively filtering the first sample data whose classification loss is greater than or equal to the first threshold;

[0160] Here, the value of the first threshold in each filtering process is successively increased to a fixed value, thereby gradually eliminating the noise sample data in the virtual sample data, and only highly similar valid virtual sample data is retained for auxiliary adversarial training.

[0161] Step S430c, when the network parameters of the first part are fixed, feature extraction is performed on the real labeled data and the virtual sample data through the backbone network to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data;

[0162] Here, the feature vectors output by the backbone network are only those of the real labeled data and the retained virtual sample data, thereby reducing unnecessary model fitting.

[0163] Step S440c: iteratively train the discriminator using the first eigenvector and the second eigenvector.

[0164] In the embodiment of the present application, in order to reduce the interference of data noise, the model is used to dynamically filter low-quality generated samples and retain high-quality video samples during the training process, and then adversarial training is performed on the continuously retained virtual sample data and a small amount of real labeled data. The trained backbone network is used to extract features from the input real labeled data and the retained virtual sample data to obtain the feature vectors of the real labeled data and the virtual sample data, and then the feature vectors are used to iteratively train the discriminator to distinguish between true and false sample data.

[0165] The training method of the above-mentioned behavior recognition model is described below in conjunction with a specific embodiment. However, it is worth noting that this specific embodiment is only for better illustrating the present application and does not constitute an improper limitation on the present application.

[0166] Anomaly detection in videos is an important problem in the field of computer vision and has a wide range of applications in the field of smart cities, such as detecting behaviors that endanger personal safety, traffic accidents, and some unusual events. However, most devices that collect video sources only record the dynamics of each moment, but do not have the ability to make automated decisions (special personnel are often required to be responsible for manual inspection). Due to the huge number of videos, it is obviously unrealistic to filter the content in the video by manpower alone. It is necessary to use computer vision and deep learning technology to automatically detect abnormal events occurring in the video.

[0167] Identifying abnormal events in online video sources is extremely difficult. Possible challenges include scarcity of labeled data due to low-probability events, large variance between / within classes, subjective differences in the definition of abnormal events, low resolution of video sources, etc. As humans, we use common sense to identify abnormalities. For example, if a crowd gathers on a street that usually has no traffic, it may be an abnormality, such as lying down or falling down. For machines, they have no common sense, only visual features. Generally speaking, the stronger the visual features, the better the expected anomaly detection performance.

[0168] For abnormal behavior detection in online video sources, it is particularly important to screen out high-quality and cheap generated data to assist the scarce real data in cold starting the model, and to further mine and train difficult samples to improve the model's fitting ability.

[0169] At the same time, for video understanding tasks, different data reading formats and inefficient pre-processing greatly restrict the efficient training of models driven by massive data. In addition, unlike low-level visual tasks such as denoising and super-resolution, high-level recognition and classification tasks are very sensitive to different decoding and pre-processing schemes. When training and deployment use different pre-processing logic, it will greatly affect the deployment accuracy. However, because different hardware platforms often provide different decoding and pre-processing implementation solutions, offline testing and online deployment are inevitably misaligned, and the sampling intervals for continuous frames of video sequences are also different. Therefore, how to improve the robustness of online deployment of video recognition models is an essential link.

[0170] For deep learning, data is undoubtedly the most important fuel, which greatly affects the training effect and applicability of the model. At present, there are still the following problems in data screening and pre-processing: how to make full use of cheap data (equivalent to virtual sample data) generated by game engines to assist in improving model performance while reducing the interference of data noise; how to efficiently perform online difficult sample mining while avoiding misfitting of noisy sample data; how to improve the deployment robustness of behavior recognition models on different hardware platforms.

[0171] The embodiment of the present application is expected to provide a dynamic training optimization and online robustness deployment algorithm for abnormal behavior detection. A large amount of data generated by Unreal Engine 4 (UE4) is used to assist a small amount of abnormal annotated data (equivalent to real annotated data) of a specific category for adversarial training. At the same time, low-quality generated samples are filtered out by online dynamic quality assessment means, and the remaining high-quality generated samples are combined with adversarial learning to perform cross-domain learning in the feature space, thereby achieving overall performance improvement. On this basis, according to the size of the model loss value, the difficult samples of the intermediate difficulty level are mined online to improve the model's fitting ability for difficult samples. In view of the differences in decoding and preprocessing implementation schemes for different hardware platforms, by adopting a variety of mixed decoding / preprocessing methods during the training process, while improving the efficient training of the model, the robustness of the model for pre-deployment processing is enhanced.

[0172] Regarding the problem of how to make full use of cheap data generated by the game engine to assist in improving the performance of the model while reducing the interference of data noise, the embodiment of the present application adopts a dynamic quality filtering method to perform adversarial training of true and false data. Figure 5A A logical flow chart of the training method provided in the embodiment of the present application, such as Figure 5A As shown, the process includes the following steps:

[0173] Step S501, obtaining a small amount of real labeled data and a large amount of virtual sample data.

[0174] The real annotated data can be real abnormal behavior data, and the virtual sample data can be data generated by the game engine. In view of the fact that the real abnormal behavior data that can be collected is scarce, the embodiment of the present application uses the game engine to generate a large amount of behavior simulation data as virtual sample data, aiming to enrich the diversity of behavior data under different scenes, different shooting angles and different lighting conditions, thereby indirectly improving the generalization performance of the behavior recognition model.

[0175] Step S502: Use the backbone network to extract features from real labeled data and virtual sample data.

[0176] Step S503: Use the discriminator to perform binary classification judgment on the features output by the backbone network to determine whether the samples are real or fake.

[0177] Among them, the features of the last layer output of the backbone network of video classification are extracted and sent to the discriminator branch. The discriminator determines whether the features extracted by the backbone network are based on the input real labeled data or the input virtual sample data.

[0178] Step S504, alternately train with the discriminator under the supervision of the classifier loss and the generator loss.

[0179] The process of alternating training can be: setting a switching frequency (e.g., switching frequency f = 2, 5, 10... times / second), when training the classifier and generator, fix the parameters of the discriminator branch; similarly, when training the discriminator, fix the parameters of the backbone network, classifier, and generator. Thus, the purpose of adversarial training is achieved. The optimization objectives are shown in formulas (1) and (2):

[0180]

[0181] Among them, L cls represents the classification loss, L adv represents the adversarial loss, λ adv is the weight of the adversarial loss. Formula (1) represents the optimization objective function obtained by weighting the classification loss and the adversarial loss. The purpose of the optimization is to maximize L adv , minimize L cls .

[0182]

[0183] Among them, L adv The value of depends on the network parameter f of the backbone network T , network parameters D of the discriminator, real labeled data X T and virtual sample data X S ; D(f T(x t )) is the discriminator for the feature vector f T (x t ) outputs the discriminant score, D(f T (x s )) is the discriminator for the feature vector f T (x s ) outputs the discriminant score, Represents data x t Sampled from real labeled data X T The expected value of the distribution function of Represents data x s Sampling from virtual sample data X S The expected value of the distribution function.

[0184] Step S505 , using the prediction scores output by the classifier to dynamically filter virtual sample data with large losses.

[0185] In addition to the domain difference problem between different source data, a large amount of noise in the virtual sample data will also directly affect the effective training of the model. In order to eliminate the noise sample data in the generated samples, retain the generated samples with high quality / features similar to the real data as much as possible, and reduce unnecessary model fitting, the embodiment of the present application proposes that when training reaches a certain stage, the generated samples with low quality / large loss are gradually filtered according to the prediction score (for example, the loss threshold is set from low to high to a fixed value in sequence). After the above step S504 is executed for several cycles, the virtual sample data with large loss is dynamically filtered using the prediction score output by the classifier, and no longer participates in the training. Thus, as the network training proceeds, only highly similar valid generated samples are retained for auxiliary adversarial training.

[0186] Figure 5B A system block diagram of a method for training a behavior recognition model provided in an embodiment of the present application; Figure 5BAs shown, the real labeled data 51b and the virtual sample data 52b are input into the backbone network 53b together; the input data is feature extracted by the backbone network 53b to obtain the spatiotemporal feature 54b; the spatiotemporal feature 54b is respectively input into the classifier 55b, the generator 56b and the discriminator 57b for adversarial training. The classifier 55b is composed of a fully connected layer, which is used to predict the category of the spatiotemporal feature 54 and output the prediction score 58b; the generator 56b is used to generate a binary generated score 59b for the feature of the virtual sample data 52b, and annotate the virtual sample data 52b with a real label, that is, to make the virtual sample data 52b as true as possible; the discriminator 57b is used to judge whether the feature output by the backbone network 53b is based on the real labeled data or the virtual sample data and output the discriminant score 60b. The structure of the discriminator 57b depends on the actual model. The embodiment of the present application uses a two-layer structure of a fully connected layer + a batch normalization layer (Batch Normalization, BN), and the loss function is the standard cross entropy of the binary classification. In the source dataset, by adjusting the weight of the adversarial loss and the frequency of alternating training through this design, better performance can be obtained than directly mixing the real labeled data and the virtual sample data for training.

[0187] During the training process, it is inevitable that some virtual sample data are more difficult to identify than other samples due to the angle, amplitude, lighting, etc. of the event, and these virtual sample data become difficult samples. At this time, in order to enhance the model's learning ability for these difficult samples, the embodiment of the present application performs online dynamic difficult sample mining during the training process. In other words, a greater weight is applied to virtual sample data with greater loss. Common means include focal loss (focalloss), online hard example mining (Online Hard Example Mining, OHEM), etc.

[0188] However, due to the objective existence of noise data, if the weight of virtual sample data is determined based on the classification loss value alone, the impact of noise data will inevitably increase. Therefore, before mining difficult samples, the embodiment of the present application divides the virtual sample data into sample difficulties based on the loss value, and only difficult samples of medium difficulty enter the mining process, thereby improving the model's ability to fit difficult samples.

[0189] In view of the differences in decoding and preprocessing implementation schemes on different hardware platforms, the embodiments of the present application adopt a variety of mixed decoding / preprocessing methods during the training process, thereby improving the efficient training of the model while enhancing the robustness of the model for pre-deployment processing. Figure 5C Several common data formats and decoding and data enhancement processing schematics provided in the embodiments of the present application; Figure 5CAs shown, for image data of different formats stored on different hardware platforms such as online parallel distributed file system 51c (lustre), distributed file system 52c (ceph), file system (petrel) 54c and object system 53c (OSG), a unified format conversion is performed through middleware 55c (summer), and then the image data in the unified format is input to the open source platform 56c (spring.data) for model training, and finally the data tool 57c (pod), data tool 58c (pttp), data tool 59c (par) and data tool 60c (pma) respectively obtain the data in the unified format to realize various recognition tasks. The embodiment of the present application provides a unified data loading interface (unified interface) through the middleware 55c for managing data, and unifies the data of different formats into the same interface mode through the middleware 55c. One possible implementation method is that the middleware 55c encapsulates all models, sample data, configuration information, etc. into a unified identification (Identity Document, ID) model, and the open source platform 56c calls the data output by the middleware 55c by calling various IDs, and uses it for general object detection, pedestrian attribute recognition and other tasks.

[0190] For the robustness of preprocessing deployment, the embodiment of the present application selects random decoding and preprocessing data enhancement schemes for the video data in a unified format, so that the model is insensitive to the differences in data decoding and preprocessing schemes, thereby improving the robustness of online deployment. Figure 5D As shown, the embodiment of the present application deploys a robust preprocessing solution on the above open source platform 56c, including: multiple encoders 51d (decoder), resize 52d (resize) and color transformation 53d (color sp), wherein: multiple encoders 51d are implemented through the image processing toolkit (Python Image Library, PIL), the open source computer vision library (Open CV), and the platform server (Kestrel), the resize 52d method includes linear (linear), cubic (cubic), nearest neighbor method (nn), weighted average (hamming), etc., and the color transformation 53d method includes image color and saturation of different storage formats (YUV420P, YUV422), red green blue format (Red Green Blue, RGB), video format (NV21), etc. At the same time, as Figure 5EAs shown, for each sample image 50e, multiple automatic data enhancement processes including colorization 51e (color), cropping 52e (crop), equalization 53e (equalize), rotation 54e (Rotate), correction 55e (gamaaj) and other operations can be deployed to enhance the data of the sample image. By randomly adopting any of the above multiple mixed decoding / preprocessing methods to train the model during the training process, the efficient training of the model is greatly improved, and the robustness of the model to pre-deployment processing is enhanced.

[0191] On the one hand, the embodiment of the present application adopts a mixed means of decoding and pre-processing implementation schemes of multiple hardware platforms for model training, which significantly improves the robustness of the model to pre-deployment processing and aligns the online and offline test processes and accuracy. On the other hand, it effectively utilizes the abnormal simulation data generated by the game engine: first, during the training process, the model is used to dynamically filter low-quality generated samples and retain high-quality video samples, and then adversarial training is performed on the continuously retained virtual sample data and a small amount of real labeled data, which improves the overall performance of the model for abnormal event detection and alleviates the dilemma of scarce abnormal data and high noise in generated data. Finally, online mining of medium-difficulty virtual sample data is also performed based on the loss value of the sample to prevent misfitting of noise data while improving the recall of truly difficult samples.

[0192] The embodiment of the present application is more inexpensive and fully utilizes the data generated by the game engine to assist in the training of the business real scene behavior recognition model, effectively alleviating the problem of data scarcity. At the same time, it improves the compatibility and robustness of the model to the decoding / preprocessing differences of different hardware platforms when it is deployed online.

[0193] The embodiments of the present application can be applied to intelligent video analysis and abnormal behavior detection in other scenarios. For example, when an abnormal behavior event occurs in the video source acquisition equipment in outdoor urban street scenes and indoor rail transit scenes, the deployed behavior detection system automatically runs the online robustness video stream test and issues an alarm, providing personnel with relevant needs with an efficient and convenient detection capability.

[0194] Based on the foregoing embodiments, the embodiments of the present application further provide a training device for a behavior recognition model, wherein the device includes the modules included, and the sub-modules and units included in each module, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0195] Figure 6 A schematic diagram of the structure of a training device for a behavior recognition model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device 600 includes a sample acquisition module 610 and a model training module 620, wherein:

[0196] The sample acquisition module 610 is used to acquire a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image;

[0197] The model training module 620 is used to use the training sample set to train a behavior recognition model built based on a generative adversarial network.

[0198] In some possible embodiments, the sample acquisition module 610 is also used to randomly select at least one decoding method and / or at least one data enhancement scheme from a preprocessing library to preprocess each of the sample images to obtain the training sample set; the preprocessing library is also used to include at least two of the decoding methods and at least two of the data enhancement schemes.

[0199] In some possible embodiments, the sample acquisition module 610 includes a decoding submodule and a data enhancement submodule, wherein: the decoding submodule is used to randomly select one of the decoding methods from the preprocessing library to perform data decoding on the at least one sample image; the data enhancement submodule is used to randomly select one of the data enhancement schemes to perform data enhancement processing on each of the decoded sample images to obtain the training sample set.

[0200] In some possible embodiments, the data enhancement scheme includes a geometric transformation operation and a color transformation operation, and the data enhancement submodule is further used to perform a random geometric transformation operation and / or a color transformation operation on each of the decoded sample images to obtain the training sample set.

[0201] In some possible embodiments, the device 600 further includes a format conversion module, which is used to perform format conversion on images in different storage formats by using a unified data loading interface to obtain the at least one sample image in a unified format.

[0202] In some possible embodiments, the training sample set includes real labeled data and generated virtual sample data; the constructed behavior recognition model includes a backbone network, a classifier, a generator and a discriminator respectively connected to the output of the backbone network; the model training module 620 includes a parameter fixing submodule and an iterative training submodule, wherein: the parameter fixing submodule is used to alternately fix the network parameters of the first part and the second part of the behavior recognition model at a specific frequency; wherein the first part includes the backbone network, the classifier and the generator, and the second part includes the discriminator; the iterative training submodule is used to iteratively train the remaining part of the behavior recognition model using the real labeled data and the virtual sample data while fixing the network parameters of one part of the behavior recognition model; wherein the one part is the first part or the second part.

[0203] In some possible embodiments, the iterative training submodule includes a feature extraction unit and a training unit, wherein: the feature extraction unit is used to perform feature extraction on the real labeled data and the virtual sample data through the backbone network while fixing the network parameters of the first part, so as to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data; the training unit is used to iteratively train the discriminator using the first feature vector and the second feature vector.

[0204] In some possible embodiments, the iterative training submodule includes a discrimination unit, a classification unit and a generation unit, wherein: the discrimination unit is used to label the feature vector output by the backbone network with a true or false label through the discriminator when the network parameters of the second part are fixed; the classification unit is used to iteratively train the classifier using the data corresponding to the feature vector carrying the true label and the data corresponding to the feature vector carrying the false label; the generation unit is used to iteratively train the generator using the data corresponding to the feature vector carrying the false label.

[0205] In some possible embodiments, the iterative training submodule includes a loss determination unit, an optimization target determination unit and a first iterative training unit, wherein: the loss determination unit is used to determine the classification loss of the classifier and the adversarial loss of the discriminator; the optimization target determination unit is used to perform a weighted sum of the classification loss and the adversarial loss to determine the optimization objective function of the behavior recognition model; the first iterative training unit is used to iteratively train the remaining part of the behavior recognition model using the real labeled data and the virtual sample data according to the optimization objective function while fixing the network parameters of a part of the behavior recognition model.

[0206] In some possible embodiments, the iterative training submodule also includes a weight adjustment unit, which is used to obtain an updated optimization objective function by adjusting the weight of the adversarial loss; the first iterative training unit is also used to iteratively train the remaining part of the behavior recognition model using the real labeled data and the virtual sample data according to the updated optimization objective function while fixing the network parameters of a part of the behavior recognition model.

[0207] In some possible embodiments, the iterative training submodule includes a sample loss unit, a sample filtering unit and a second iterative training unit, wherein: the sample loss unit is used to determine the classification loss of the virtual sample data passing through the classifier; the sample filtering unit is used to iteratively filter the first sample data whose classification loss is greater than or equal to a first threshold; wherein the value of the first threshold is successively increased to a fixed value in each filtering process; and the second iterative training unit is used to iteratively train the remaining part of the behavior recognition model using the real labeled data and the filtered virtual sample data while fixing the network parameters of a part of the behavior recognition model.

[0208] In some possible embodiments, the second iterative training unit includes a determination subunit, an adjustment subunit and a training subunit, wherein: the determination subunit is used to use the filtered virtual sample data as the second sample data; the adjustment subunit is used to increase the weight of the second sample data when the classification loss of the second sample data is greater than or equal to a second threshold; wherein the second threshold is less than the first threshold; the training subunit is used to iteratively train the remaining part of the behavior recognition model using the real labeled data and the second sample data while fixing the network parameters of a part of the behavior recognition model.

[0209] In some possible embodiments, the adjustment subunit is further used to determine hard-to-separate sample data of an intermediate difficulty level from the second sample data as new second sample data based on the classification loss of the second sample data; and to increase the weight of the new second sample data when the classification loss of the new second sample data is greater than a second threshold.

[0210] In some possible embodiments, the device also includes a data acquisition module, a preprocessing module and a behavior recognition module, wherein: the data acquisition module is used to acquire original video stream data; the preprocessing module is used to preprocess the original video stream data to obtain a video sequence to be recognized; the behavior recognition module is used to use the trained behavior recognition model to recognize the video sequence to be recognized and determine abnormal behavior in the video sequence to be recognized.

[0211] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0212] It should be noted that in the embodiment of the present application, if the training method of the above-mentioned behavior recognition model is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0213] Correspondingly, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the training method of the behavior recognition model described in any of the above embodiments are implemented.

[0214] Correspondingly, in an embodiment of the present application, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip is running, it is used to implement the steps in the training method of the behavior recognition model described in any of the above embodiments.

[0215] Correspondingly, in an embodiment of the present application, a computer program product is also provided. When the computer program product is executed by a processor of an electronic device, it is used to implement the steps in the training method of the behavior recognition model described in any of the above embodiments.

[0216] Based on the same technical concept, an embodiment of the present application provides an electronic device for implementing the training method of the behavior recognition model recorded in the above method embodiment. Figure 7 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device 700 includes a memory 710 and a processor 720, wherein the memory 710 stores a computer program that can be executed on the processor 720, and when the processor 720 executes the program, the steps in the training method of the behavior recognition model described in any embodiment of the present application are implemented.

[0217] The memory 710 is configured to store instructions and applications executable by the processor 720, and can also cache data to be processed or processed by the processor 720 and various modules in the electronic device (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0218] When the processor 720 executes the program, the steps of any of the above-mentioned methods for training a behavior recognition model are implemented. The processor 720 generally controls the overall operation of the electronic device 700.

[0219] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0220] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various electronic devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0221] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0222] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0223] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms. The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present application.

[0225] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be used as a unit separately, or two or more units can be integrated into one unit; the above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product, which is stored in a storage medium, including a number of instructions for the automatic test line of the device to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0226] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain a new method embodiment or device embodiment.

[0227] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for training a behavior recognition model, characterized in that: The method comprises: Acquire a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image; the training sample set includes real annotated data and generated virtual sample data; Using the training sample set, training a behavior recognition model built based on a generative adversarial network; the built behavior recognition model includes a backbone network, a classifier, a generator, and a discriminator respectively connected to the output of the backbone network; The method of using the training sample set to train a behavior recognition model based on a generative adversarial network includes: Alternately fixing network parameters of a first part and a second part of the behavior recognition model; wherein the first part includes the backbone network, the classifier and the generator, and the second part includes the discriminator; While fixing the network parameters of a part of the behavior recognition model, the remaining part of the behavior recognition model is iteratively trained using the real labeled data and the virtual sample data; wherein the part is the first part or the second part.

2. The method according to claim 1, characterized in that The step of obtaining a training sample set includes: At least one decoding method and / or at least one data enhancement scheme is randomly selected from a preprocessing library to preprocess each of the sample images to obtain the training sample set; the preprocessing library includes at least two of the decoding methods and at least two of the data enhancement schemes.

3. The method according to claim 2, characterized in that The randomly selecting at least one decoding method and at least one data enhancement scheme from the preprocessing library to preprocess each of the sample images to obtain the training sample set includes: Randomly select one of the decoding methods from the preprocessing library to perform data decoding on each of the sample images; A data enhancement scheme is randomly selected to perform data enhancement processing on each of the decoded sample images to obtain the training sample set.

4. The method according to claim 3, characterized in that The data enhancement scheme includes a geometric transformation operation and a color transformation operation, and the randomly selecting one of the data enhancement schemes performs data enhancement processing on each of the decoded sample images to obtain the training sample set, including: A random geometric transformation operation and / or color transformation operation is performed on each of the decoded sample images to obtain the training sample set.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The images in different storage formats are converted by using a unified data loading interface to obtain the at least one sample image in a unified format.

6. The method according to claim 1, characterized in that The method of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the generated virtual sample data while fixing the network parameters of a part of the behavior recognition model comprises: Under the condition that the network parameters of the first part are fixed, feature extraction is performed on the real labeled data and the virtual sample data through the backbone network to obtain a first feature vector of the real labeled data and a second feature vector of the virtual sample data; The discriminator is iteratively trained using the first feature vector and the second feature vector.

7. The method according to claim 6, characterized in that The method of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the generated virtual sample data while fixing the network parameters of a part of the behavior recognition model comprises: When the network parameters of the second part are fixed, the feature vector output by the backbone network is labeled with a true or false label by the discriminator; Iteratively training the classifier using data corresponding to feature vectors carrying true labels and data corresponding to feature vectors carrying false labels; The generator is iteratively trained using the data corresponding to the feature vector carrying the false label.

8. The method according to claim 1, characterized in that The method of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the generated virtual sample data while fixing the network parameters of a part of the behavior recognition model comprises: Determining a classification loss of the classifier and an adversarial loss of the discriminator; Performing a weighted summation on the classification loss and the adversarial loss to determine an optimization objective function of the behavior recognition model; According to the optimization objective function, while fixing a portion of the network parameters in the behavior recognition model, the remaining portion of the behavior recognition model is iteratively trained using the real labeled data and the virtual sample data.

9. The method according to claim 8, characterized in that The method further comprises: By adjusting the weight of the adversarial loss, an updated optimization objective function is obtained; The step of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the virtual sample data according to the optimization objective function while fixing a part of the network parameters in the behavior recognition model comprises: According to the updated optimization objective function, while fixing a portion of the network parameters in the behavior recognition model, the remaining portion of the behavior recognition model is iteratively trained using the real labeled data and the virtual sample data.

10. The method according to claim 1, characterized in that The method of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the generated virtual sample data while fixing the network parameters of a part of the behavior recognition model comprises: Determine the classification loss of the virtual sample data through the classifier; Iteratively filter the first sample data whose classification loss is greater than or equal to a first threshold; wherein the value of the first threshold is increased to a fixed value in turn during each filtering process; When a part of the network parameters of the behavior recognition model is fixed, the remaining part of the behavior recognition model is iteratively trained using the real labeled data and the filtered virtual sample data.

11. The method according to claim 10, characterized in that The method of iteratively training the remaining part of the behavior recognition model by using the real labeled data and the filtered virtual sample data while fixing a part of the network parameters of the behavior recognition model comprises: Using the filtered virtual sample data as second sample data; In the case where the classification loss of the second sample data is greater than or equal to a second threshold, increasing the weight of the second sample data; wherein the second threshold is less than the first threshold; While fixing a portion of the network parameters of the behavior recognition model, the remaining portion of the behavior recognition model is iteratively trained using the real labeled data and the second sample data.

12. The method according to claim 11, characterized in that When the classification loss of the second sample data is greater than or equal to a second threshold, increasing the weight of the second sample data includes: According to the classification loss of the second sample data, determining hard-to-separate sample data of an intermediate difficulty level from the second sample data as new second sample data; When the classification loss of the new second sample data is greater than a second threshold, the weight of the new second sample data is increased.

13. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Get the original video stream data; Preprocessing the original video stream data to obtain a video sequence to be identified; The video sequence to be identified is identified using a trained behavior recognition model to determine abnormal behaviors in the video sequence to be identified.

14. A training device for a behavior recognition model, characterized in that: The device comprises a sample acquisition module and a model training module, wherein: The sample acquisition module is used to acquire a training sample set; the training sample set is obtained by hybrid decoding and / or data enhancement processing of at least one sample image; the training sample set includes real annotated data and generated virtual sample data; The model training module is used to train a behavior recognition model built based on a generative adversarial network using the training sample set; the built behavior recognition model includes a backbone network, a classifier, a generator and a discriminator respectively connected to the output of the backbone network; The model training module is used to alternately fix the network parameters of the first part and the second part of the behavior recognition model; wherein the first part includes the backbone network, the classifier and the generator, and the second part includes the discriminator; when the network parameters of one part of the behavior recognition model are fixed, the remaining part of the behavior recognition model is iteratively trained using the real labeled data and the virtual sample data; wherein the one part is the first part or the second part.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps in the method according to any one of claims 1 to 13 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program, comprising a computer readable code, wherein when the computer readable code is executed in an electronic device, a processor in the electronic device executes a program configured to implement the method according to any one of claims 1 to 13.

18. A computer program product, comprising one or more instructions, wherein the one or more instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 13.

Images