Event detection method and device, and event detection model training method and device

Abstract

Embodiments of the present specification provide a detection model training method and device, and an event detection method and device. The detection model training method comprises receiving a sample video containing an event, and extracting a sample feature of the sample video; inputting the sample feature into a first category module to generate a first category sequence of the sample video, and inputting the sample feature into a second category module to generate a second category sequence of the sample video; obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a detection model training method. One or more embodiments of the present specification also relate to an event detection method, a detection model training apparatus, an event detection apparatus, a computing device, and a computer-readable storage medium. BACKGROUND

[0002] In current urban brain projects such as public security, municipal, and traffic, a large amount of camera video data is generated every day, and how to effectively mine information from the video data is a problem to be solved. Visual analysis based on a single picture cannot effectively utilize the effective information in time sequence, and it is difficult to obtain good results in a video scene. Therefore, the current common practice is to rely on post-supervision, that is, when someone reports an abnormal behavior event (such as theft, fighting, etc.) occurs, the corresponding period of video is retrieved to view the situation at that time. This post-supervision method can only be used for post-evidence, and cannot discover abnormal situations in real time and make warnings and interventions, and the existing detection model can only identify abnormal events with obvious action features, and can easily miss abnormal events with non-obvious action features.

[0003] Therefore, there is an urgent need to provide a detection model training method that can detect abnormal events with non-obvious action features. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a detection model training method. One or more embodiments of the present specification also relate to an event detection method, a detection model training apparatus, an event detection apparatus, a computing device, and a computer-readable storage medium to solve the technical defects in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a detection model training method is provided, including:

[0006] receiving a sample video containing an event, and extracting a sample feature of the sample video;

[0007] inputting the sample feature into a first category module to generate a first category sequence of the sample video, and inputting the sample feature into a second category module to generate a second category sequence of the sample video;

[0008] obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function.

[0009] According to a second aspect of the embodiments of the present specification, a detection model training method is provided, including:

[0010] displaying a video input interface for the user based on a calling request of the user;

[0011] receiving a sample video containing an event sent by the user based on the video input interface, and extracting sample features of the sample video;

[0012] inputting the sample features into a first category module to generate a first category sequence of the sample video, and inputting the sample features into a second category module to generate a second category sequence of the sample video;

[0013] obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function.

[0014] According to a third aspect of the embodiments of the present specification, a detection model training method is provided, comprising:

[0015] receiving a calling request sent by a user, wherein the calling request carries a sample video containing an event;

[0016] extracting sample features of the sample video;

[0017] inputting the sample features into a first category module to generate a first category sequence of the sample video, and inputting the sample features into a second category module to generate a second category sequence of the sample video;

[0018] obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function.

[0019] According to a fourth aspect of the embodiments of the present specification, an event detection method is provided, comprising:

[0020] receiving a video containing an event, and inputting the video into a detection model to obtain an event contained in the video and a time when the event occurs, wherein the detection model is obtained by training according to the detection model training method described above;

[0021] generating a corresponding early warning strategy for the event in a case where the event meets a preset early warning condition.

[0022] According to a fifth aspect of the embodiments of the present specification, an event detection method is provided, applied to a city management scenario, comprising:

[0023] receiving a video containing a public security event, and inputting the video into a detection model to obtain a public security event contained in the video and a time when the public security event occurs, wherein the detection model is obtained by training according to the detection model training method described above;

[0024] When the public security incident meets the preset warning conditions, a corresponding warning strategy is generated for the public security incident.

[0025] According to a sixth aspect of the embodiments of this specification, an event detection method is provided, which is applied to an offline retail scenario, including:

[0026] Receiving a video containing a violation event, and inputting the video into a detection model to obtain the violation event contained in the video and the time when the violation event occurred, wherein the detection model is trained according to the detection model training method as described above;

[0027] When the violation event meets the preset warning conditions, a corresponding warning strategy is generated for the violation event.

[0028] According to a seventh aspect of the embodiments of this specification, a detection model training device is provided, comprising:

[0029] A first receiving module is configured to receive a sample video containing an event and extract sample features of the sample video;

[0030] A first generating module is configured to input the sample features into a first classification module to generate a first classification sequence of the sample video, and input the sample features into a second classification module to generate a second classification sequence of the sample video;

[0031] The first training module is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0032] According to an eighth aspect of the embodiments of this specification, a detection model training device is provided, comprising:

[0033] An interface display module is configured to display a video input interface to the user based on a user's call request;

[0034] a second receiving module configured to receive a sample video containing an event sent by the user based on the video input interface, and extract sample features of the sample video;

[0035] A second generating module is configured to input the sample features into the first classification module to generate a first classification sequence of the sample video, and input the sample features into the second classification module to generate a second classification sequence of the sample video;

[0036] The second training module is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0037] According to a ninth aspect of an embodiment of the present specification, a detection model training apparatus is provided, comprising:

[0038] A request receiving module configured to receive a calling request sent by a user, wherein the calling request carries a sample video containing an event;

[0039] A feature extraction module configured to extract a sample feature of the sample video;

[0040] A third generation module configured to input the sample feature into a first category module to generate a first category sequence of the sample video, and input the sample feature into a second category module to generate a second category sequence of the sample video;

[0041] A third training module configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0042] According to a tenth aspect of an embodiment of the present specification, an event detection apparatus is provided, comprising:

[0043] An event detection module configured to receive a video containing an event, and input the video into a detection model to obtain the event contained in the video and a time when the event occurs, wherein the detection model is obtained by training according to the detection model training method described above;

[0044] A strategy generation module configured to generate a corresponding early warning strategy for the event if the event meets a preset early warning condition.

[0045] According to an eleventh aspect of an embodiment of the present specification, an event detection apparatus applied to a city management scenario is provided, comprising:

[0046] A second event detection module configured to receive a video containing a public security event, and input the video into a detection model to obtain the public security event contained in the video and a time when the public security event occurs, wherein the detection model is obtained by training according to the detection model training method described above;

[0047] A second strategy generation module configured to generate a corresponding early warning strategy for the public security event if the public security event meets a preset early warning condition.

[0048] According to a twelfth aspect of an embodiment of the present specification, an event detection apparatus applied to an offline retail scenario is provided, comprising:

[0049] a third event detection module configured to receive a video containing a violation event, and input the video into a detection model to obtain the violation event contained in the video and a time when the violation event occurs, wherein the detection model is trained according to the detection model training method as described above;

[0050] a third strategy generation module configured to generate a corresponding early warning strategy for the violation event if the violation event meets preset early warning conditions.

[0051] According to a thirteenth aspect of an embodiment of the present specification, a computing device is provided, comprising:

[0052] a memory and a processor;

[0053] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the detection model training method or the steps of the event detection method.

[0054] According to a fourteenth aspect of an embodiment of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which, when executed by a processor, implement the steps of the detection model training method or the steps of the event detection method.

[0055] One embodiment of the present specification implements a detection model training method and device, and an event detection method and device. The detection model training method comprises receiving a sample video containing an event, and extracting a sample feature of the sample video; inputting the sample feature into a first category module to generate a first category sequence of the sample video, and inputting the sample feature into a second category module to generate a second category sequence of the sample video; obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function; specifically, the second category sequence generated by the second category module assists the first category sequence generated by the first category module, and the detection model trained by the detection model training method can not only detect events with obvious action features through the first category module, but also cluster features based on the distribution of events in the feature space through the second category module, train a projection vector to mine events with non-obvious action features, so as to comprehensively and accurately detect events in a video. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is an example diagram of a specific application scenario of an event detection method provided by one embodiment of the present specification;

[0057] Figure 2is a flowchart of a first detection model training method provided by an embodiment of the present specification;

[0058] Figure 3 is a schematic diagram of clustering events in a sample video using a feature projection method in a detection model training method provided by an embodiment of the present specification;

[0059] Figure 4 is a flowchart of a first event detection method provided by an embodiment of the present specification;

[0060] Figure 5 is a structural schematic diagram of a detection model in an event detection method provided by an embodiment of the present specification;

[0061] Figure 6 is a flowchart of a second detection model training method provided by an embodiment of the present specification;

[0062] Figure 7 is a flowchart of a third detection model training method provided by an embodiment of the present specification;

[0063] Figure 8 is a structural schematic diagram of a first detection model training device provided by an embodiment of the present specification;

[0064] Figure 9 is a structural schematic diagram of a second detection model training device provided by an embodiment of the present specification;

[0065] Figure 10 is a structural schematic diagram of a third detection model training device provided by an embodiment of the present specification;

[0066] Figure 11 is a structural schematic diagram of a first event detection device provided by an embodiment of the present specification;

[0067] Figure 12 is a flowchart of a second event detection method provided by an embodiment of the present specification;

[0068] Figure 13 is a flowchart of a third event detection method provided by an embodiment of the present specification;

[0069] Figure 14 is a structural schematic diagram of a second event detection device provided by an embodiment of the present specification;

[0070] Figure 15 is a structural schematic diagram of a third event detection device provided by an embodiment of the present specification;

[0071] Figure 16 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0072] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. However, the present description can be practiced without the specific details, and can also be practiced with multiple different implementations than those described herein. Thus, the present description is not limited to the implementations described herein but can be practiced with alterations and modifications in the implementations discussed.

[0073] The terminology used in this description of one or more embodiments shall be only used for the purpose of describing particular embodiments and is not intended to be limiting of one or more embodiments. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0074] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first entity discussed below could later be discussed as a second entity, and similarly, a second entity discussed below could later be discussed as a first entity without departing from the scope of one or more embodiments. As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to a determination" or "in response to the fact that...."

[0075] First, the noun terms related to one or more embodiments of the present description are explained.

[0076] CNN: Convolution Neural Network, a kind of convolution neural network containing convolution calculation and having deep structure, is a kind of feedforward neural network.

[0077] MIL: Multi Instance Learning, multi-instance learning.

[0078] CAS: Class Activation Sequence, class activation sequence.

[0079] t-SNE: t-distributed stochastic neighbor embedding, a kind of machine learning algorithm for dimension reduction, is one of the algorithms with better effect in visualization algorithm.

[0080] THUMOS-14 dataset: contains a large number of human actions in real environment open source video.

[0081] ActivityNet dataset: is the largest dataset for current temporal action detection task, covering 200 different daily activities.

[0082] IoU: is a standard for measuring the accuracy of detecting corresponding objects in a specific dataset.

[0083] In the current peace, municipal, traffic and other urban brain projects, a large amount of camera video data will be generated every day, how to effectively mine information from these video data is a problem to be solved. Based on single picture visual analysis cannot effectively use the effective information in time sequence, it is difficult to obtain good effect in video scene. Therefore, the current common method is to rely on post supervision, that is, when someone reports an abnormal behavior event (such as theft, fighting, etc.), the corresponding period of video is retrieved to view the situation at that time. This way of post supervision can only be used for post evidence, and cannot find abnormal situations in real time and make warning and intervention. Therefore, based on computer vision technology, real-time analysis of abnormal behavior in urban scene has significant practical significance.

[0084] Therefore, the detection model training method provided in the present specification is provided. One or more embodiments of the present specification also relate to an event detection method, a detection model training device, an event detection device, a computing device, and a computer readable storage medium, which are described in detail in the following embodiments.

[0085] When the detection model training method provided in the present specification is applied to analyze abnormal behavior (fighting, arson, theft, etc.) in urban scene, deep learning technology can be used to analyze the picture taken by the camera in real time, analyze whether there is abnormal behavior, and once found, issue a warning and intervene, which is beneficial to prevent further expansion of the harm in time; In practical application, the detection model training method can not only be applied to analyze abnormal behavior in urban scene, but also be applied to other scenes of detecting action, for example, analyzing sports behavior in sports scene, using the detection model provided in the present specification to extract basketball, swimming, football, running and other sports actions in the monitoring video of a certain square, the crowd nearby can be determined to be more inclined to a certain sport through analysis of the sports action, so that sports facilities can be increased in the square in a targeted manner, and the outdoor experience of users can be enhanced.

[0086] In practical applications, the abnormal action (such as the abnormal behavior described above) segment detection method specifically refers to inputting a long video, identifying the time period of abnormal action occurrence through the abnormal action segment detection method, and returning the starting and ending time of the action and the specific category of the action. The strong supervision based abnormal action detection method needs a large amount of manual annotation data to train the network, including the start and end time of the abnormal action in the video and the category of the action, and the high manual annotation cost greatly reduces the practicability of the strong supervision based abnormal action detection method.

[0087] Compared with strong supervision, the weak supervision based abnormal action detection algorithm only needs to manually mark whether the whole video contains abnormal actions such as fighting, falling, stealing, etc. in the training process, without marking the start and end time of the action. Therefore, the weak supervision based abnormal action detection algorithm greatly reduces the annotation workload and is more practical.

[0088] In specific implementation, the weak supervision based abnormal action detection can be trained through a multi instance learning (MIL) based method to train the network model, and the specific details are as follows:

[0089] For the input video, first, high-level semantic features are extracted based on a feature extractor, and the extracted video features are marked as X R T×D , where T represents the time dimension feature, and D represents the action feature dimension of each time point. Then, a two-layer time sequence 1-dimensional convolutional network is used to generate a class activation sequence (CAS), which is represented as follows:

[0090]

[0091] , where f conv represents a 1-dimensional convolutional network, and φ1 and φ2 represent network parameters. The generated A R T×C is a class activation sequence, that is, for each action feature, the corresponding action category probability value is output, where T represents the length of the video in the time dimension, that is, the time dimension, and C is the number of categories of abnormal actions. For example, if the video is 3 seconds, the length of the video in the time dimension is 1 second, and the video includes three abnormal actions, that is, three categories, then the generated class activation sequence can obtain the probability of each second video belonging to the three categories, such as the probability of belonging to playing basketball, swimming and running between 0 seconds and 1 second. A 3*3 matrix, that is, a class activation sequence, is generated.

[0092] Since there is no time-series annotation information for action clips during training, multi-instance learning is usually used to train network parameters. Specifically, for each action category, the corresponding video action probability (the probability that the video clip contains this action) is calculated based on the generated category activation sequence A. The formula is as follows:

[0093]

[0094] That is, for category C, by selecting k act The number with the highest probability Then ask The average of the corresponding video action probability a c .

[0095] After obtaining the video action probability for each action category, the network model is trained based on the cross entropy loss function:

[0096]

[0097] Among them, N is the number of samples input in each training, C is the number of abnormal action categories, y=[y1,y2,...,y C ]∈R C It is the video-level action category label provided during training, that is, if action c appears in the video, the corresponding y c is 1 if the value is set, otherwise it is 0.

[0098] Multi-instance learning relies on the cross-entropy loss function generated by action category classification to reversely optimize the entire model parameters. Since classification and detection are two different tasks—the classification task aims to achieve accurate classification based on salient features, while the detection task aims to locate and classify complete action segments—weakly supervised learning methods that rely on multi-instance learning can typically only locate salient regions, and may miss less obvious abnormal actions. For example, for abnormal actions such as fighting, fights with large movements and obvious features are easy to locate and detect, but fights with small movements and less obvious features are often overlooked. Therefore, detection models trained solely through multi-instance learning can only locate some abnormal actions, and the overall effect is not very good.

[0099] See also Figure 1 , Figure 1 An example diagram showing a specific application scenario of an event detection method provided by an embodiment of this specification is shown.

[0100] In the embodiments of this specification, the event detection method is specifically introduced by taking the application of analyzing abnormal behaviors in urban scenarios as an example.

[0101] Figure 1The application scenarios of the event detection method provided in the embodiments of the present specification include an image acquisition terminal 102, an image receiving terminal 104, and a server 106. Specifically, the image receiving terminal 104 receives a monitoring video a collected in real time by the image acquisition terminal 102 under a city scene; after receiving the video a, the image receiving terminal 104 sends the video a to the server 106, and after receiving the video a, the server 106 inputs the video a into a detection model. In the case where the detection model detects an abnormal behavior (fighting, arson, theft, etc.) existing in the video a, a warning is issued and intervention is performed, which is conducive to timely preventing further expansion of the harm and maintaining social order.

[0102] In actual application, the detection model is obtained by training based on the related class activation module and the feature projection module. The detection model can not only detect an abnormal behavior with obvious action features, that is, with high saliency, but also detect an abnormal behavior with unobvious action features.

[0103] The event detection method provided in the embodiments of the present specification is applied in analyzing abnormal behaviors under a city scene. The detection model obtained by training based on the related class activation module and the feature projection module can comprehensively and accurately obtain abnormal behaviors in a monitoring video, so that a response measure can be quickly made based on the abnormal behaviors to improve user experience.

[0104] Referring to Figure 2 , Figure 2 A flowchart of a first detection model training method according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0105] Step 202: Receive a sample video containing an event, and extract sample features of the sample video.

[0106] The application scenarios of the detection model training method are different, and the events contained in the sample video are also different. For example, if the detection model training method is applied in analyzing abnormal behaviors under a city scene, the event can be understood as a motion event such as fighting, theft, or arson. If the detection model training method is applied in analyzing motion types under a sports scene, the event can be understood as a motion event such as swimming, playing basketball, playing football, or running. For ease of understanding, the detection model training method is applied in analyzing abnormal behaviors under a city scene in the embodiments of the present specification.

[0107] Specifically, the extraction of the sample features of the sample video includes:

[0108] Extracting high-dimensional sample features of the sample video according to a preset feature extractor, wherein the high-dimensional sample features include time dimension features and event dimension features at each time point.

[0109] The preset feature extractor can be set according to actual application, and the specification does not make any limitation on this. For example, commonly used feature extractors include I3D, C3D, P3D, and the like.

[0110] In a specific implementation, the feature extractor is pre-trained, and in the embodiments of the specification, the high-dimensional features of the sample video can be directly extracted based on the pre-trained parameters; specifically, for the received sample video containing events, first, the high-dimensional sample features of each sample video are extracted based on the pre-trained feature extractor, and then the high-dimensional sample features are recorded as X e R T×D wherein T represents the time dimension feature, and D represents the event dimension feature at each time point.

[0111] In the embodiments of the specification, before the detection model is trained based on the sample video containing events, the high-dimensional features of the sample video are quickly extracted based on the pre-trained feature extractor, and subsequently, the detection model can be trained based on the accurate high-dimensional features, so as to improve the speed and accuracy of the detection model training.

[0112] Step 204: inputting the sample features into the first category module to generate the first category sequence of the sample video, and inputting the sample features into the second category module to generate the second category sequence of the sample video.

[0113] Specifically, after the high-dimensional sample features X of the sample video are extracted, the high-dimensional sample features X are input into the first category module and the second category module respectively, to generate the first category sequence of the sample video and the second category sequence of the sample video, wherein the first category module and the second category module can be understood as two convolutional networks of the entire detection model. For example, if the detection model is a convolutional neural network, the first category module and the second category module are two parallel convolutional network layers of the convolutional neural network.

[0114] Specifically, the inputting of the sample features into the first category module to generate the first category sequence of the sample video includes:

[0115] The sample features are input into the first category module, and the first category sequence of the sample video generated by the first category module through the multi-instance learning method is obtained.

[0116] In actual application, the first category module can be a related category activation module. After the high-dimensional sample features X are obtained, the high-dimensional sample features X are input into the related category activation module, and the related category activation module generates the category activation sequence A of the sample video by using the multi-instance learning method. Through the multi-instance learning method, the salient events with obvious action features in the sample video can be accurately located.

[0117] Further, the sample feature is input into the second category module to generate a second category sequence of the sample video.

[0118] The sample feature is input into the second category module, and the second category sequence of the sample video generated by the second category module through the feature projection method is obtained.

[0119] In actual application, the second category module can be a feature projection module composed of projection vectors. The projection vectors in the feature projection module are pre-trained before use. After obtaining the high-dimensional sample feature X, the high-dimensional sample feature X is input into the feature projection module, and the feature projection module generates the category activation sequence S of the sample video based on the determined projection vectors. The feature projection module can be based on the clustering characteristics of the distribution of actions (i.e., events) in the feature space, so that relatively accurate positioning can also be achieved for abnormal actions with unclear features.

[0120] The feature projection module using the feature projection method in the embodiments of the present specification assists the related category activation module using the multi-instance learning method to train the detection model, so that the detection model obtained by training can realize the detection of events with obvious action features and events with unclear action features.

[0121] Still taking the abnormal actions including fighting, arson, theft and the like as examples, for each type of action, the distribution thereof in the feature space has clustering characteristics.

[0122] Referring to Figure 3 , Figure 3 FIG. 1 shows a schematic diagram of clustering of events in a sample video using a feature projection method according to one embodiment of the present specification.

[0123] Taking the events in Figure 3 as examples, it can be seen from the visualization method based on t-SNE that the features of fighting (Fight) and the features of not fighting (no fight) are separable in the feature space, i.e., fighting and not fighting can be distinguished by finding a reasonable projection direction without additional supervision information. As shown in Figure 3 , the conventional multi-instance learning method can only be positioned to the salient regions, and the feature projection method can more completely discover all the actions of this type, so that the effect is obviously improved.

[0124] Based on such a discovery, for each type of abnormal action c, a corresponding projection vector is learned to separate the target action. The specific learning method is to learn a reasonable projection vector by training a projection loss function:

[0125]

[0126] wherein, N is the number of input sample videos at each time of training, T is the length in the time dimension of a single video, C is the number of categories of abnormal actions, y i，c is 1 when the Cth category of action exists in the video, and 0 otherwise, p c is the projection vector corresponding to the category C, and X i，k is the action feature; by minimizing the feature projection loss function, the corresponding projection vector can be learned for each specific abnormal action category, so as to realize the mining and positioning of the category of action.

[0127] In specific implementation, the first category module is a relevant category activation module, and the second category module is a feature projection module. After the high-dimensional sample feature X is input into the relevant category activation module and the feature projection module, the relevant category activation module generates the category activation sequence A of the sample video by using the method of multiple instance learning. Since the method of multiple instance learning can only locate the significant area, the feature projection module is used as an auxiliary. Specifically, the feature projection module is composed of a projection vector. The feature projection module uses the projection vector inside which is pre-trained to generate the category activation sequence S of the sample video based on the determined projection vector after the high-dimensional sample feature X is input into the feature projection module. The feature projection module is based on the clustering characteristics of the distribution of actions (i.e. events) in the feature space, so it can also accurately locate the abnormal action with unobvious features. Therefore, the detection model of the embodiments of the present specification can accurately detect events with both obvious action features and unobvious action features after the first category module and the second category module are trained.

[0128] Step 206: obtaining a target loss function based on the first category sequence and the second category sequence, and training the detection model according to the target loss function.

[0129] Specifically, after the first category sequence and the second category sequence are obtained, a target loss function of the detection model is obtained based on the first category sequence and the second category sequence, and then the network parameters of the detection model are adjusted according to the target loss function to realize the training of the detection model.

[0130] Specifically, the obtaining of the target loss function based on the first category sequence and the second category sequence comprises:

[0131] obtaining a first loss function of the first category module based on the first category sequence, and obtaining a second loss function of the second category module based on the second category sequence;

[0132] obtaining the target loss function based on the first loss function and the second loss function.

[0133] Specifically, in the training process of the detection model, the network parameters of the entire detection model are trained end-to-end, that is, the first category module and the second category module are trained simultaneously.

[0134] Therefore, the target loss function of the detection model can be accurately obtained through the first loss function of the first category module and the second loss function of the second category module.

[0135] In specific implementation, the obtaining of the target loss function based on the first loss function and the second loss function includes:

[0136] The first loss function and the second loss function are weighted, and the weighted first loss function and the weighted second loss function are added to obtain the target loss function.

[0137] In the above example, still taking the first category module as the relevant category activation module and the second category module as the feature projection module as an example, the relevant category activation module and the feature projection module are trained simultaneously in the training of the detection model. The loss function based on the relevant category activation module and the feature projection module can obtain the loss function of the entire detection model. The loss function of the relevant category activation module is obtained through the category activation sequence A, the loss function of the feature projection module is obtained through the category activation sequence S, and specifically, the calculation formula of the loss function L activation of the relevant category activation module is shown in the above formula 3, and the calculation formula of the loss function L project of the feature projection module is shown in the above formula 4. Then, the target loss function of the detection model is obtained by adding the loss functions generated by the relevant category activation module and the feature projection module, and is specifically shown as follows:

[0138] L = L activation + ρL project Formula 5

[0139] Wherein, ρ is used to control the relative weight of the two loss functions, and is usually 0.01.

[0140] In the embodiments of the present specification, the target loss function of the detection model is obtained by weighting and adding the first loss function of the first category module and the second loss function of the second category module. Based on the target loss function, the network parameters of the detection model are adjusted, so that the trained detection model can detect events in the video more comprehensively and accurately in subsequent use.

[0141] In another embodiment of the present specification, after the sample feature is input into the first category module to generate the first category sequence of the sample video, the method further comprises:

[0142] The method based on the attention mechanism corrects the first category sequence to obtain a corrected first category sequence.

[0143] In practical applications, in addition to being able to increase the performance of the detection model by excavating the non-significant area in the sample video through the feature projection module, the overall effect of the detection model can also be improved based on the attention mechanism. The attention mechanism can optimize the category activation sequence by fusing and comparing the correlation between each action feature. This method is usually used on the basis of multiple instance learning to further improve the performance of the detection model.

[0144] Specifically, the method for correcting the first category sequence based on the attention mechanism is as follows:

[0145]

[0146] wherein A i and are the category activation sequence before and after correction (i.e. the first category sequence), F i is the intermediate semantic feature generated by a 1-dimensional convolutional network after receiving X∈R R×D as input. For specific details, refer to formula 1 above. The category activation sequence A is corrected based on the cosine similarity between features, which can more easily find abnormal action intervals outside the significant area.

[0147] In the embodiment of the present specification, the method based on the attention mechanism corrects the category activation sequence generated by the related category activation module of multiple instance learning, and the feature projection method uses the effective information that the distribution of actions in the feature space has clustering to simultaneously train the detection model, which can further improve the performance of the detection model obtained by training. In addition, the second category module can also be trained by the method of attention mechanism instead of the feature projection method, which can also improve the performance of the detection model as a whole, that is, without using the second category module of the feature projection method to assist the first category module, only using the method of attention mechanism to correct the category activation sequence generated by the first category module, so as to achieve the method for detecting non-significant events in the video.

[0148] In the embodiments of the present specification, the first category module in the detection model training method can be regarded as a related category activation module based on a multi-instance learning method, and the second category module can be regarded as a feature projection module. When the detection model is trained by using the detection model training method, not only can the first category module be used to locate the abnormal events with obvious action features, but also the second category module can be used to cluster features based on the distribution of events in the feature space, train the projection vector, and mine the abnormal events with non-obvious action features, so that comprehensive and accurate positioning and detection of all abnormal events in the video can be realized.

[0149] Referring to Figure 4 , Figure 4 A flowchart of a first event detection method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0150] Step 402: receiving a video containing an event, inputting the video into a detection model, and obtaining the event contained in the video and the time when the event occurs, wherein the detection model is obtained by training according to the detection model training method as described above.

[0151] The application scenarios of the event detection method are different, and the events contained in the video are also different. For example, if the event detection method is applied to analyze abnormal behaviors in a city scene, the event can be understood as a fighting, theft, arson, or other action event. If the event detection method is applied to analyze the types of sports in a sports scene, the event can be understood as swimming, playing basketball, playing football, running, or other action events.

[0152] Specifically, after inputting the video into the detection model, a plurality of events contained in the video and the start and end time of each event can be obtained.

[0153] Step 404: generating a corresponding early warning strategy for the event if the event meets a preset early warning condition.

[0154] Optionally, the receiving of the video containing the event includes:

[0155] Based on the calling request of the user, a video input interface is displayed for the user, and a video containing an event sent by the user based on the video input interface is received.

[0156] Optionally, the receiving of the video containing the event includes:

[0157] A calling request sent by a user is received, wherein the calling request carries a video containing an event.

[0158] The preset warning condition is set according to actual application, and the specification does not make any limitation on this, for example, the event detection method is applied in analyzing abnormal behaviors in a city scene, the warning condition can include arson and robbery, and when the detected event in the video contains an event of arson and / or robbery, it means that the event meets the preset warning condition, and at this time, a corresponding warning strategy is generated for the event, such as manual intervention or shouting warning, etc.

[0159] The event detection method provided by the embodiment of the specification can realize comprehensive, accurate and real-time prediction of events in a video through a detection model trained according to a related category activation module of a multi-instance learning method and a feature projection module of a feature projection method, and then generate a reasonable warning strategy according to the predicted event, thereby improving user experience.

[0160] Referring to Figure 5 , Figure 5 The structure of the detection model in an event detection method provided by an embodiment of the specification is shown.

[0161] Specifically, when training the detection model in Figure 5 , the video obtained according to time is input into the feature extractor, the feature extractor extracts the features of the input video, extracts the high-dimensional video features of the video, and then inputs the high-dimensional video features into the related category activation module to generate the category activation sequence A, and simultaneously inputs the high-dimensional video features into the feature projection module to generate the category activation sequence S, and then the feature projection module clusters the actions (such as fighting, theft, robbery, etc. Figure 5 in ) in the video, and then generates the loss function of the related category activation module based on the category activation sequence A, wherein the category activation sequence A obtains the cross-entropy loss function of the related category activation module through the action aggregation module, and simultaneously generates the loss function of the feature projection module based on the category activation sequence S, and finally calculates the loss function of the detection model according to the loss function of the related category activation module and the loss function of the feature projection module, and adjusts the network parameters of the detection model based on the loss function of the detection model to realize the training of the detection model.

[0162] In actual application, after the video is input into the detection model, it will also be input into the related category activation module after high-dimensional feature extraction to generate the category activation sequence A and input into the feature projection module to generate the category activation sequence S, and then the category activation sequence A and the category activation sequence S are weighted and fused to obtain a new category activation sequence, and then the new category activation sequence can be used to locate more complete abnormal actions, thereby improving the recall rate and accuracy of abnormal actions.

[0163] Specifically, the category activation sequences generated by the related category activation module and the feature projection module are fused, and are expressed as follows:

[0164] A final = A + γS Formula 7

[0165] Wherein, γ is a hyperparameter for adjusting the weight of the two category activation sequences, and is usually 0.5 in actual landing process.

[0166] The detection model in the event detection method provided by the embodiments of the present specification is obtained by training according to the related category activation module of the multi-instance learning method and the feature projection module of the feature projection method. In the subsequent detection of events in the video by the detection model, the events in the video can be more comprehensively, accurately and real-time detected, and due to the fusion of the new category activation sequence in the detection model, the recall rate and accuracy of the events in the video can be greatly improved.

[0167] In order to verify the performance of the detection model obtained by training, the classical model STPN (Sparse Temporal Pooling Network) is taken as a reference on the data set THUMOS-14 and ActivityNet, and the feature projection method and the attention mechanism method are added respectively, and the indicators are mAP (mean average precision) under different IoU (Intersection over Union). The comparison results are shown in the following table:

[0168] Table 1: Comparison experiment after introducing the attention mechanism method and the feature projection method on the THUMOS-14 data set

[0169]

[0170]

[0171] Table 2: Comparison experiment after introducing the attention mechanism method and the feature projection method on the ActivityNet data set

[0172] Model name IoU=0.5 IoU=0.75 IoU=0.95 Average STPN 29.3 16.9 2.6 17.3 STPN+attention 32.1 19.4 4.2 20.1 STPN+feature projection 34.0 21.9 5.1 21.9

[0173] As shown in Table 1 and Table 2, after introducing the attention mechanism method and the feature projection method, each indicator on the THUMOS-14 and ActivityNet data sets is significantly improved; and compared with STPN+attention, the indicators of STPN+feature projection are more significantly improved, which verifies the effectiveness of the detection model training method of the present specification.

[0174] Referring to Figure 6 , Figure 6A flowchart of a second detection model training method provided by an embodiment of the present specification is shown, and specifically includes the following steps.

[0175] Step 602: Present a video input interface to the user based on a call request of the user.

[0176] Step 604: Receive a sample video containing an event sent by the user based on the video input interface, and extract sample features of the sample video.

[0177] Step 606: Input the sample features into a first category module to generate a first category sequence of the sample video, and input the sample features into a second category module to generate a second category sequence of the sample video.

[0178] Step 608: Obtain a target loss function based on the first category sequence and the second category sequence, train the detection model according to the target loss function, and return the detection model to the user.

[0179] The detection model training method provided by the embodiment of the present specification assists the first category sequence generated by the first category module with the second category sequence generated by the second category module, so that the detection model trained by the detection model training method can not only detect events with obvious action features through the first category module, but also cluster features based on the distribution of events in the feature space through the second category module, train a projection vector to mine events with non-obvious action features, so as to realize comprehensive and accurate detection of events in a video.

[0180] The above is a schematic scheme of the second detection model training method of the present embodiment. It should be noted that the technical scheme of the second detection model training method belongs to the same concept as the technical scheme of the first detection model training method described above, and the details of the technical scheme of the second detection model training method not described in detail can be referred to the description of the technical scheme of the first detection model training method.

[0181] Referring to Figure 7 , Figure 7 A flowchart of a third detection model training method provided by an embodiment of the present specification is shown, and specifically includes the following steps.

[0182] Step 702: Receive a call request sent by a user, wherein the call request carries a sample video containing an event.

[0183] Step 704: Extract sample features of the sample video.

[0184] Step 706: inputting the sample feature into the first category module to generate a first category sequence of the sample video, and inputting the sample feature into the second category module to generate a second category sequence of the sample video.

[0185] Step 708: obtaining a target loss function based on the first category sequence and the second category sequence, training the detection model according to the target loss function, and returning the detection model to the user.

[0186] The detection model training method provided by the embodiments of the present specification can assist the first category sequence generated by the first category module by the second category sequence generated by the second category module. The detection model trained by the detection model training method can not only detect events with obvious action features by the first category module, but also cluster features based on the distribution of events in the feature space by the second category module, train a projection vector, and mine events with non-obvious action features, so as to comprehensively and accurately detect events in a video.

[0187] The above is a schematic scheme of the third detection model training method of the present embodiment. It should be noted that the technical scheme of the third detection model training method belongs to the same concept as the technical scheme of the first detection model training method described above. The details of the technical scheme of the third detection model training method that are not described in detail can be referred to the description of the technical scheme of the first detection model training method.

[0188] Corresponding to the above method embodiments, the present specification also provides detection model training device embodiments, Figure 8 A structural schematic diagram of a first detection model training device provided by an embodiment of the present specification is shown. As shown in the figure, Figure 8 The device comprises:

[0189] A first receiving module 802 is configured to receive a sample video containing an event and extract a sample feature of the sample video.

[0190] A first generating module 804 is configured to input the sample feature into the first category module to generate a first category sequence of the sample video, and input the sample feature into the second category module to generate a second category sequence of the sample video.

[0191] A model training module 806 is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0192] Optionally, the first receiving module 802 is further configured to:

[0193] extract high-dimensional sample features of the sample video according to a preset feature extractor, wherein the high-dimensional sample features comprise time-dimension features and event-dimension features at each time point.

[0194] Optionally, the first generation module 804 is further configured to:

[0195] input the sample features into a first category module, and obtain a first category sequence of the sample video generated by the first category module through a multi-instance learning method.

[0196] Optionally, the first generation module 804 is further configured to:

[0197] input the sample features into a second category module, and obtain a second category sequence of the sample video generated by the second category module through a feature projection method.

[0198] Optionally, the first training module 806 is further configured to:

[0199] obtain a first loss function of the first category module based on the first category sequence, and obtain a second loss function of the second category module based on the second category sequence;

[0200] obtain the target loss function based on the first loss function and the second loss function.

[0201] Optionally, the apparatus further comprises:

[0202] a correction module configured to correct the first category sequence based on an attention mechanism method to obtain a corrected first category sequence.

[0203] Optionally, the first training module 806 is further configured to:

[0204] weight the first loss function and the second loss function, add the weighted first loss function and the second loss function to obtain the target loss function.

[0205] The detection model training apparatus provided by the embodiments of the present specification can assist the first category sequence generated by the first category module through the second category sequence generated by the second category module, and realize the detection model trained through the detection model training method. The detection model can not only detect events with obvious action features through the first category module, but also cluster features based on the distribution of events in the feature space through the second category module, train a projection vector to mine events with non-obvious action features, and thus can realize comprehensive and accurate detection of events in a video.

[0206] The above is a schematic scheme of the first detection model training device of the embodiment. It should be noted that the technical scheme of the first detection model training device belongs to the same concept as the technical scheme of the first detection model training method described above. Details of the technical scheme of the first detection model training device that are not described in detail can be seen from the description of the technical scheme of the first detection model training method.

[0207] Corresponding to the method embodiments described above, the present specification also provides detection model training device embodiments, Figure 9 The structure of the second detection model training device provided by an embodiment of the present specification is shown in a schematic diagram. As shown in the figure, Figure 9 The device comprises:

[0208] The interface display module 902 is configured to display a video input interface for the user based on the user's call request;

[0209] The second receiving module 904 is configured to receive a sample video containing an event sent by the user based on the video input interface, and extract sample features of the sample video;

[0210] The second generation module 906 is configured to input the sample features into the first category module to generate a first category sequence of the sample video, and input the sample features into the second category module to generate a second category sequence of the sample video;

[0211] The second training module 908 is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0212] The detection model training device provided by the embodiment of the present specification assists the first category sequence generated by the first category module with the second category sequence generated by the second category module, realizes the detection model trained by the detection model training method, and can not only detect events with obvious action features through the first category module, but also cluster features based on the distribution of events in the feature space through the second category module, train projection vectors to realize the mining of events with non-obvious action features, so as to realize comprehensive and accurate detection of events in the video.

[0213] The above is a schematic scheme of the second detection model training device of the embodiment. It should be noted that the technical scheme of the second detection model training device belongs to the same concept as the technical scheme of the second detection model training method described above. Details of the technical scheme of the second detection model training device that are not described in detail can be seen from the description of the technical scheme of the second detection model training method.

[0214] Corresponding to the method embodiments described above, the specification also provides an event detection device embodiment, Figure 10 A structural diagram of a third event detection device provided by an embodiment of the specification is shown. As shown in the figure, Figure 10 The device comprises:

[0215] The request receiving module 1002 is configured to receive a calling request sent by a user, wherein the calling request carries a sample video containing an event;

[0216] The feature extraction module 1004 is configured to extract sample features of the sample video;

[0217] The third generation module 1006 is configured to input the sample features into a first category module to generate a first category sequence of the sample video, and input the sample features into a second category module to generate a second category sequence of the sample video;

[0218] The third training module 1008 is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

[0219] The detection model training device provided by the embodiment of the specification is assisted by the second category sequence generated by the second category module to the first category sequence generated by the first category module, and the detection model trained by the detection model training method can not only detect events with obvious action features through the first category module, but also cluster features based on the distribution of events in the feature space through the second category module, train the projection vector to implement the mining of events with non-obvious action features, so as to realize comprehensive and accurate detection of events in the video.

[0220] The above is a schematic scheme of the third detection model training device of the embodiment. It should be noted that the technical scheme of the third detection model training device belongs to the same concept as the technical scheme of the third detection model training method described above, and the details of the technical scheme of the third detection model training device that are not described in detail can be referred to the description of the technical scheme of the third detection model training method.

[0221] Corresponding to the method embodiments described above, the specification also provides an event detection device embodiment, Figure 11 A structural diagram of an event detection device provided by an embodiment of the specification is shown. As shown in the figure, Figure 11 The device comprises:

[0222] The event detection module 1102 is configured to receive a video containing an event, and input the video into a detection model to obtain the event contained in the video and a time when the event occurs, wherein the detection model is trained according to the detection model training method as described above.

[0223] The policy generation module 1104 is configured to generate a corresponding early warning policy for the event if the event meets a preset early warning condition.

[0224] Optionally, the event detection module 1102 is further configured to:

[0225] display a video input interface for a user based on a calling request of the user, and receive a video containing an event sent by the user based on the video input interface.

[0226] Optionally, the event detection module 1102 is further configured to:

[0227] receive a calling request sent by a user, wherein the calling request carries a video containing an event.

[0228] The event detection device provided by the embodiments of the present specification can realize comprehensive, accurate and real-time prediction of events in a video through a detection model trained according to a relevant category activation module and a feature projection module of a multi-instance learning method, and then generate a reasonable early warning policy according to the predicted event, thereby improving user experience.

[0229] The above is a schematic scheme of the event detection device of the present embodiment. It should be noted that the technical scheme of the event detection device belongs to the same concept as the technical scheme of the event detection method described above, and the details of the technical scheme of the event detection device that are not described in detail can be referred to the description of the technical scheme of the event detection method.

[0230] Referring to Figure 12 , Figure 12 A flowchart of a second event detection method provided by an embodiment of the present specification is shown, wherein the method is applied to a city management scenario, and specifically includes the following steps.

[0231] Step 1202: receiving a video containing a public security event, and inputting the video into a detection model to obtain the public security event contained in the video and a time when the public security event occurs, wherein the detection model is trained according to the detection model training method as described above.

[0232] The public security event includes but is not limited to action events such as fighting, theft and brawling in the city management scenario.

[0233] Step 1204: generating a corresponding early warning strategy for the public security event in a case where the public security event meets a preset early warning condition.

[0234] The preset early warning condition is set according to actual application, for example, the preset early warning condition is that the public security event is a fire event, and the early warning strategy is generated.

[0235] The event detection method provided in the specification embodiment is applied to a city management scene, can realize comprehensive, accurate and real-time prediction of the acquired public security event in the city management scene by using a detection model trained by a related category activation module and a feature projection module of a multi-instance learning method, and then generates a reasonable early warning strategy according to the predicted public security event, so as to better maintain the safety of city management.

[0236] The above is a schematic scheme of the second event detection method of the embodiment. It should be noted that the technical scheme of the event detection method belongs to the same concept as the technical scheme of the first event detection method described above, and the details of the technical scheme of the event detection method that are not described in detail can be referred to the description of the technical scheme of the first event detection method.

[0237] Referring to Figure 13 , Figure 13 A flowchart of a third event detection method provided by one embodiment of the specification is shown, wherein the method is applied to an offline retail scene, and specifically includes the following steps.

[0238] Step 1302: receiving a video containing a violation event, and inputting the video into a detection model to obtain a violation event contained in the video and a time when the violation event occurs, wherein the detection model is obtained by training according to the detection model training method as described above.

[0239] The violation event includes but is not limited to action events such as theft and damage to sold goods in an offline retail scene.

[0240] Step 1304: generating a corresponding early warning strategy for the violation event in a case where the violation event meets a preset early warning condition.

[0241] The preset early warning condition is set according to actual application, for example, the preset early warning condition is that the violation event is a damage to sold goods event, and the early warning strategy is generated.

[0242] The event detection method provided by the embodiment of the present specification is applied to an offline retail scene, and a detection model trained according to a related category activation module and a feature projection module of a feature projection method can be used to comprehensively, accurately and in real time predict a violation event in the offline retail scene, and then a reasonable early warning strategy is generated according to the predicted violation event, so as to avoid loss of various resources and articles in the offline retail scene.

[0243] The above is a schematic scheme of the third event detection method of the embodiment. It should be noted that the technical scheme of the event detection method belongs to the same concept as the technical scheme of the first event detection method, and the details of the technical scheme of the event detection method that are not described in detail can be referred to the description of the technical scheme of the first event detection method.

[0244] Referring to Figure 14 , Figure 14 Fig. 1 shows a structural schematic diagram of a second event detection device provided by an embodiment of the present specification, wherein the device is applied to a city management scene and includes:

[0245] The second event detection module 1402 is configured to receive a video containing a public security event, and input the video into a detection model to obtain the public security event contained in the video and a time when the public security event occurs, wherein the detection model is obtained by training according to the detection model training method as described above.

[0246] The second strategy generation module 1404 is configured to generate a corresponding early warning strategy for the public security event if the public security event meets a preset early warning condition.

[0247] The event detection device provided by the embodiment of the present specification is applied to a city management scene, and a detection model trained according to a related category activation module and a feature projection module of a feature projection method can be used to comprehensively, accurately and in real time predict a public security event in the city management scene, and then a reasonable early warning strategy is generated according to the predicted public security event, so as to better maintain the safety of city management.

[0248] The above is a schematic scheme of the second event detection device of the embodiment. It should be noted that the technical scheme of the event detection device belongs to the same concept as the technical scheme of the second event detection method, and the details of the technical scheme of the event detection device that are not described in detail can be referred to the description of the technical scheme of the second event detection method.

[0249] Referring to Figure 15 , Figure 15A structural diagram of a third event detection device provided by an embodiment of the present specification is shown, wherein the device is applied to an offline retail scene, and the device comprises:

[0250] A third event detection module 1502 is configured to receive a video containing a violation event, and input the video into a detection model to obtain the violation event contained in the video and a time when the violation event occurs, wherein the detection model is obtained by training according to the detection model training method as described above.

[0251] A third strategy generation module 1504 is configured to generate a corresponding early warning strategy for the violation event in a case where the violation event meets a preset early warning condition.

[0252] The event detection device provided by the embodiment of the present specification is applied to an offline retail scene, and can realize comprehensive, accurate and real-time prediction of the violation event in the offline retail scene by using the detection model obtained by training according to the relevant category activation module and the feature projection module of the multi-instance learning method, and then generate a reasonable early warning strategy according to the predicted violation event, so as to avoid the loss of various resources and goods in the offline retail scene.

[0253] The above is a schematic scheme of the third event detection device of the present embodiment. It should be noted that the technical scheme of the event detection device belongs to the same concept as the technical scheme of the third event detection method described above, and the details of the technical scheme of the event detection device that are not described in detail can be referred to the description of the technical scheme of the third event detection method.

[0254] Figure 16 A structural block diagram of a computing device 1600 according to an embodiment of the present specification is shown. The components of the computing device 1600 include but are not limited to a memory 1610 and a processor 1620. The processor 1620 is connected to the memory 1610 through a bus 1630, and a database 1650 is used to save data.

[0255] The computing device 1600 also includes an access device 1640 that enables the computing device 1600 to communicate via one or more networks 1660. Examples of such networks include public switched telephone networks (PSTNs), local area networks (LANs), wide area networks (WANs), personal area networks (PAN), or combinations of such networks, such as the Internet. The access device 1640 can include one or more of any type of network interface (for example, network interface cards (NICs)) such as IEEE 802.11 wireless local area network (WLAN) wireless interfaces, Worldwide Interoperability Microwave Access (Wi-MAX) interfaces, Ethernet interfaces, Universal Serial Bus (USB) interfaces, cellular network interfaces, Bluetooth interfaces, near-field communication (NFC) interfaces, and the like.

[0256] In an embodiment of the present specification, the above-described components of the computing device 1600 and other components not shown in FIG. 16 can be connected to each other through a bus, for example. Figure 16 It should be understood that the above-described components of the computing device 1600 can be connected to each other through a bus, for example. Figure 16 The computing device structure diagram shown is merely for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0257] The computing device 1600 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a PC. The computing device 1600 can also be a mobile or stationary server.

[0258] The processor 1620 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the detection model training method or the steps of the event detection method.

[0259] The above is a schematic scheme of a computing device according to an embodiment of the present specification. It should be noted that the technical scheme of the computing device belongs to the same concept as the technical scheme of the detection model training method or the event detection method described above, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the detection model training method or the event detection method.

[0260] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the detection model training method or the steps of the event detection method.

[0261] The above is a schematic solution of the computer readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the detection model training method or the event detection method described above belong to the same concept, and the technical solution of the storage medium which is not described in detail can be referred to the description of the technical solution of the detection model training method or the event detection method.

[0262] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different than the order in which they are recited in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0263] The computer instructions include computer program code, which can be in the form of source code, object code, executable code, or some intermediate form. The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0264] It should be noted that for the foregoing method embodiments, in order to facilitate description, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the embodiments of the present specification are not limited by the order of the described actions, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily necessary for the embodiments of the present specification.

[0265] In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0266] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. Alternative embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and variations of the embodiments of the present specification can be made in light of the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is limited only by the claims and their full scope and equivalents.

Claims

1. A detection model training method, comprising: Receiving a sample video containing an event, and extracting sample features of the sample video; Inputting the sample features into a first classification module and using a multi-instance learning method to generate a first category sequence of the sample video; and inputting the sample features into a second classification module and generating a second category sequence of the sample video based on clustering characteristics of event distribution in a feature space; A target loss function is obtained based on the first category sequence and the second category sequence, and the detection model is trained according to the target loss function.

2. The detection model training method according to claim 1, wherein extracting sample features of the sample video comprises: The high-dimensional sample features of the sample video are extracted according to a preset feature extractor, wherein the high-dimensional sample features include time dimension features and event dimension features at each time point.

3. The detection model training method according to claim 1, wherein inputting the sample features into a second category module to generate a second category sequence of the sample video comprises: The sample features are input into a second category module, and a second category sequence of the sample video generated by the second category module through a feature projection method is obtained.

4. The detection model training method according to claim 1, wherein obtaining a target loss function based on the first category sequence and the second category sequence comprises: Obtaining a first loss function of the first category module based on the first category sequence, and obtaining a second loss function of the second category module based on the second category sequence; The target loss function is obtained based on the first loss function and the second loss function.

5. The detection model training method according to claim 4, further comprising: The first category sequence is corrected based on an attention mechanism method to obtain a corrected first category sequence.

6. The detection model training method according to claim 5, wherein obtaining the target loss function based on the first loss function and the second loss function comprises: The first loss function and the second loss function are weighted, and the weighted first loss function and the second loss function are added to obtain the target loss function.

7. A detection model training method comprising: Displaying a video input interface to the user based on the user's call request; receiving a sample video containing an event sent by the user based on the video input interface, and extracting sample features of the sample video; Inputting the sample features into a first classification module and using a multi-instance learning method to generate a first category sequence of the sample video; and inputting the sample features into a second classification module and generating a second category sequence of the sample video based on clustering characteristics of event distribution in a feature space; A target loss function is obtained based on the first category sequence and the second category sequence, the detection model is trained according to the target loss function, and the detection model is returned to the user.

8. A detection model training method, comprising: receiving a calling request sent by a user, wherein the calling request carries a sample video containing an event; extracting sample features of the sample video; inputting the sample features into a first category module to generate a first category sequence of the sample video using a multi-instance learning method, and inputting the sample features into a second category module to generate a second category sequence of the sample video based on clustering characteristics of distribution of the event in a feature space; obtaining a target loss function based on the first category sequence and the second category sequence, training the detection model according to the target loss function, and returning the detection model to the user.

9. An event detection method, comprising: receiving a video containing an event, inputting the video into a detection model to obtain the event contained in the video and a time at which the event occurs, wherein the detection model is trained according to the detection model training method of any one of claims 1-6; generating a corresponding early warning strategy for the event in a case where the event meets a preset early warning condition.

10. The event detection method of claim 9, wherein the receiving a video containing an event comprises: exhibiting a video input interface to a user based on a calling request of the user, and receiving a video containing an event sent by the user based on the video input interface.

11. The event detection method of claim 9, wherein the receiving a video containing an event comprises: receiving a calling request sent by a user, wherein the calling request carries a video containing an event.

12. An event detection method applied to a city management scenario, comprising: receiving a video containing a public security event, inputting the video into a detection model to obtain the public security event contained in the video and a time at which the public security event occurs, wherein the detection model is trained according to the detection model training method of any one of claims 1-6; generating a corresponding early warning strategy for the public security event in a case where the public security event meets a preset early warning condition.

13. An event detection method applied to an offline retail scenario, comprising: receiving a video containing a violation event, inputting the video into a detection model to obtain the violation event contained in the video and a time at which the violation event occurs, wherein the detection model is trained according to the detection model training method of any one of claims 1-6; generating a corresponding early warning strategy for the violation event in a case where the violation event meets a preset early warning condition.

14. A detection model training apparatus, comprising: a first receiving module configured to receive a sample video containing an event and extract sample features of the sample video; a first generating module configured to input the sample features into a first category module to generate a first category sequence of the sample video using a multi-instance learning method, and input the sample features into a second category module to generate a second category sequence of the sample video based on clustering characteristics of distribution of the event in a feature space; The first training module is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

15. A detection model training device comprising: An interface display module is configured to display a video input interface to the user based on a user's call request; a second receiving module configured to receive a sample video containing an event sent by the user based on the video input interface, and extract sample features of the sample video; a second generation module configured to input the sample features into the first classification module and generate a first category sequence of the sample video using a multi-instance learning method, and to input the sample features into the second classification module and generate a second category sequence of the sample video based on clustering characteristics of event distribution in the feature space; The second training module is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

16. A detection model training device comprising: A request receiving module is configured to receive a call request sent by a user, wherein the call request carries a sample video containing an event; A feature extraction module is configured to extract sample features of the sample video; a third generation module configured to input the sample features into the first classification module and generate a first category sequence of the sample video using a multi-instance learning method, and input the sample features into the second classification module and generate a second category sequence of the sample video based on clustering characteristics of event distribution in the feature space; The third training module is configured to obtain a target loss function based on the first category sequence and the second category sequence, and train the detection model according to the target loss function.

17. An event detection device, comprising: a first event detection module configured to receive a video containing an event, input the video into a detection model, and obtain the event contained in the video and the time when the event occurred, wherein the detection model is trained according to the detection model training method according to any one of claims 1 to 6; The first strategy generating module is configured to generate a corresponding warning strategy for the event when the event meets a preset warning condition.

18. An event detection device, applied to urban management scenarios, comprising: A second event detection module is configured to receive a video containing a public security event, input the video into a detection model, and obtain the public security event contained in the video and the time when the public security event occurred, wherein the detection model is trained according to the detection model training method according to any one of claims 1 to 6; The second strategy generating module is configured to generate a corresponding warning strategy for the public security event when the public security event meets the preset warning conditions.

19. An event detection device, used in offline retail scenarios, comprising: a third event detection module configured to receive a video containing a violation event, and input the video into a detection model to obtain the violation event contained in the video and a time when the violation event occurs, wherein the detection model is trained according to the detection model training method in any one of claims 1-6; a third strategy generation module configured to generate a corresponding early warning strategy for the violation event if the violation event meets a preset early warning condition.

20. A computing device comprising: a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which, when executed by the processor, implement the steps of the detection model training method in any one of claims 1-6, 7, 8 and the steps of the event detection method in any one of claims 9-11, 12, 13.

21. A computer readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the detection model training method in any one of claims 1-6, 7, 8 and the steps of the event detection method in any one of claims 9-11, 12, 13.

Images