Training of a target detection model, target detection method, device and medium

Abstract

The present disclosure provides a target detection model training method, a target detection method, an apparatus and a storage medium. The training method comprises: generating training samples based on video samples and augmented samples; using an encoder module and based on video feature information corresponding to the training samples, generating query feature information and constructing a first loss function corresponding to the augmented samples; using a decoder module and based on the query feature information, generating first classification confidence information corresponding to the video samples and regression information used to represent target positions, second classification confidence information corresponding to the augmented samples, and constructing a second loss function corresponding to the training samples; and using the first loss function and the second loss function for adjustment processing. The present disclosure can enhance the distinguishability of the input video features, increase the feature difference of the action and improve the distinguishability of the features; the classification training of the model is sufficient, so that the prediction result is more accurate.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a target detection model training method, a target detection method, a device and a storage medium. BACKGROUND

[0002] With the increasing amount of video data, the demand for analysis and processing of video data is increasing. For example, in the scenarios of live content security detection and short video dangerous action detection, a video action detection method is needed to identify risk actions in video data. Currently, when performing action detection, a DETR (Bidirectional Encoder Representations from Transformer, based on transformer structure) model is usually used for target detection. The DETR model uses the structure of the Transformer to realize query-based two-dimensional image target detection. The Transformer structure is a network structure based on the attention mechanism. By constructing a model through the Transformer, the performance of the video action detection method can be effectively improved. In the process of implementing the present application, the inventors found that in the training of the DETR model, each input video segment obtains the same number of positive samples as the number of labels by matching, and other additional predictions are set as negative samples, so that the number of positive samples participating in the training is insufficient, causing insufficient classification training of the DETR model and low classification accuracy. SUMMARY

[0003] Therefore, one of the technical problems to be solved by the present application is to provide a target detection model training method, a target detection method, a device and a storage medium.

[0004] According to a first aspect of the present disclosure, a target detection model training method is provided, wherein the target detection model comprises an encoder module and a decoder module, and the training method comprises: obtaining an enhanced sample corresponding to a video sample, generating a training sample based on the video sample and the enhanced sample; using the encoder module and based on video feature information corresponding to the training sample, generating query feature information and constructing a first loss function corresponding to the enhanced sample; using the decoder module and based on the query feature information, generating first classification confidence information corresponding to the video sample and regression information for representing target positions, second classification confidence information corresponding to the enhanced sample, and constructing a second loss function corresponding to the training sample; and using the first loss function and the second loss function to adjust the target detection model.

[0005] Optionally, the enhanced sample comprises a positive sample and a negative sample corresponding to the video sample; and the constructing the first loss function corresponding to the enhanced sample comprises: obtaining segment feature information corresponding to the positive sample and the negative sample based on the video feature information; and generating the first loss function according to the segment feature information.

[0006] Optionally, the encoder module comprises a fully connected layer and a region of interest pooling (RoI Pooling) layer; and the obtaining the segment feature information corresponding to the positive sample and the negative sample based on the video feature information comprises: processing the video feature information through the fully connected layer to obtain fully connected feature corresponding to the video feature information; and performing feature extraction processing on the fully connected feature through the RoI Pooling layer to obtain the segment feature information.

[0007] Optionally, the generating the first loss function according to the segment feature information comprises: obtaining a first segment feature corresponding to the video sample and a second segment feature corresponding to the positive sample in the segment information; determining a first sample feature based on the first segment feature and the second segment feature; obtaining a third segment feature corresponding to the negative sample in the segment information; determining a second sample feature based on the first segment feature, the second segment feature and the third segment feature; and generating the first loss function according to the first sample feature and the second sample feature.

[0008] Optionally, the constructing the second loss function corresponding to the training sample comprises: determining a first classification loss function corresponding to the first classification confidence information; determining a second classification loss function corresponding to the second classification confidence information; and generating the second loss function based on the second classification loss function and the second classification loss function.

[0009] Optionally, the first classification loss function comprises a first cross-entropy loss function; the second classification loss function comprises a second cross-entropy loss function; and the generating the second loss function based on the second classification loss function and the second classification loss function comprises: taking a sum of the first cross-entropy loss function and the second cross-entropy loss function as the second loss function.

[0010] Optionally, a first action type label of the video sample is determined; and the positive sample corresponding to the video sample is obtained from other videos based on the first action type label; and a second action type label of the positive sample is the same as the first action type label.

[0011] Optionally, based on the first action type label, the negative sample corresponding to the video sample is obtained from other videos; wherein the second action type label of the positive sample is different from the first action type label.

[0012] Optionally, a first video segment corresponding to the first action type label is determined; a video segment is extracted from the video sample as the negative sample based on the first video segment; wherein the ratio of the overlapping length of the first video segment and the second video segment to the length sum is less than a preset intersection over union threshold.

[0013] Optionally, the training sample is processed using a preset backbone network to generate the video feature information; wherein the backbone network comprises a neural network model.

[0014] Optionally, the encoder module comprises an encoder based on a Transformer structure; and the decoder module comprises a decoder based on a Transformer structure.

[0015] According to a second aspect of the present disclosure, a target detection method is provided, comprising: obtaining a trained target detection model; wherein the target detection model is trained by the training method as above; using the target detection model and based on a video to be detected, generating classification confidence information corresponding to the video to be detected, and regression information for representing target position.

[0016] According to a third aspect of the present disclosure, a training device of a target detection model is provided, wherein the target detection model comprises an encoder module and a decoder module; the training device comprises: a sample generation module, configured to obtain an enhanced sample corresponding to a video sample, and generate a training sample based on the video sample and the enhanced sample; an encoding processing module, configured to use the encoder module and based on video feature information corresponding to the training sample, generate query feature information, and construct a first loss function corresponding to the enhanced sample; a decoding processing module, configured to use the decoder module and based on the query feature information, generate first classification confidence information corresponding to the video sample and regression information for representing target position, second classification confidence information corresponding to the enhanced sample, and construct a second loss function corresponding to the training sample; and a model adjustment module, configured to use the first loss function and the second loss function to perform adjustment processing on the target detection model.

[0017] Optionally, the enhanced sample comprises positive samples and negative samples corresponding to the video sample; the encoding processing module comprises: a segment feature acquisition unit configured to acquire segment feature information corresponding to the positive samples and the negative samples based on the video feature information; and a first loss determination unit configured to generate a first loss function according to the segment feature information.

[0018] Optionally, the encoder module comprises a fully connected layer and a region of interest pooling (RoI Pooling) layer; the segment feature acquisition unit is configured to acquire fully connected features corresponding to the video feature information by processing the video feature information through the fully connected layer, and acquire the segment feature information by performing feature extraction processing on the fully connected features through the RoI Pooling layer.

[0019] Optionally, the first loss determination unit is configured to acquire first segment features corresponding to the video sample and second segment features corresponding to the positive samples in the segment information, determine first sample features based on the first segment features and the second segment features, acquire third segment features corresponding to the negative samples in the segment information, determine second sample features based on the first segment features, the second segment features and the third segment features, and generate the first loss function according to the first sample features and the second sample features.

[0020] Optionally, the decoding processing module comprises: a classification loss determination unit configured to determine a first classification loss function corresponding to the first classification confidence information and determine a second classification loss function corresponding to the second classification confidence information; and a second loss determination unit configured to generate the second loss function based on the second classification loss function and the second classification loss function.

[0021] Optionally, the first classification loss function comprises a first cross-entropy loss function, the second classification loss function comprises a second cross-entropy loss function, and the second loss determination unit is specifically configured to take a sum of the first cross-entropy loss function and the second cross-entropy loss function as the second loss function.

[0022] Optionally, the sample generation module comprises a positive sample acquisition unit configured to determine a first action type label of the video sample, and acquire the positive samples corresponding to the video sample from other videos based on the first action type label, wherein a second action type label of the positive samples is the same as the first action type label.

[0023] Optionally, the sample generation module comprises a negative sample acquisition unit configured to acquire the negative sample corresponding to the video sample from other videos based on the first action type label, wherein the second action type label of the positive sample is different from the first action type label.

[0024] Optionally, the negative sample acquisition unit is configured to determine a first video segment corresponding to the first action type label, and extract a video segment from the video sample as the negative sample based on the first video segment, wherein a ratio of an overlapping length of the first video segment and the second video segment to a length sum is less than a preset intersection over union threshold.

[0025] Optionally, the feature information acquisition module is configured to process the training sample using a preset backbone network to generate the video feature information, wherein the backbone network comprises a neural network model.

[0026] Optionally, the encoder module comprises a Transformer structure-based encoder, and the decoder module comprises a Transformer structure-based decoder.

[0027] According to a fourth aspect of the present disclosure, a training device of an object detection model is provided, comprising a memory and a processor coupled to the memory, the processor being configured to execute the training method as described above based on instructions stored in the memory.

[0028] According to a fifth aspect of the present disclosure, an object detection device is provided, comprising a model acquisition module configured to acquire a trained object detection model, wherein the object detection model is trained by the training method as described above; and a detection processing module configured to use the object detection model and generate classification confidence information corresponding to a to-be-detected video and regression information for representing a target position based on the to-be-detected video.

[0029] According to a sixth aspect of the present disclosure, an object detection device is provided, comprising a memory and a processor coupled to the memory, the processor being configured to execute the object detection method as described above based on instructions stored in the memory.

[0030] According to a seventh aspect of the present disclosure, a computer readable storage medium is provided, the computer readable storage medium storing computer instructions, the instructions being executed by a processor to perform the method as described above.

[0031] The training method of the target detection model, the target detection method, the device and the storage medium of the present disclosure can generate training samples based on video samples and enhanced samples and construct a loss function corresponding to the enhanced samples, can enhance the distinguishability of the input video features, improve the feature similarity of the same action, can increase the feature difference of different actions and improve the distinguishability of the features; generate classification confidence information corresponding to the video samples and the enhanced samples based on the query feature information and construct a loss function, increase the number of positive samples and other training samples, fully train the model for classification, and make the prediction result more accurate; and improve the user's use experience. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 Flowchart of one embodiment of the training method of the target detection model according to the present disclosure;

[0034] Figure 2 Flowchart of constructing the first loss function in one embodiment of the training method of the target detection model according to the present disclosure;

[0035] Figure 3 Flowchart of generating the first loss function according to the segment feature information in one embodiment of the training method of the target detection model according to the present disclosure;

[0036] Figure 4 Network framework diagram of the target detection model of the present disclosure;

[0037] Figure 5 Diagram of the decoder in the target detection model of the present disclosure;

[0038] Figure 6 Flowchart of constructing the second loss function in one embodiment of the training method of the target detection model according to the present disclosure;

[0039] Figure 7 Diagram of the decoder in the target detection model of the present disclosure;

[0040] Figure 8 Flowchart of one embodiment of the target detection method according to the present disclosure;

[0041] Figure 9Module schematic diagram of one embodiment of a training device for a target detection model according to the present disclosure;

[0042] Figure 10 Module schematic diagram of another embodiment of a training device for a target detection model according to the present disclosure;

[0043] Figure 11 Module schematic diagram of an encoding processing module in one embodiment of a training device for a target detection model according to the present disclosure;

[0044] Figure 12 Module schematic diagram of a decoding processing module in one embodiment of a training device for a target detection model according to the present disclosure;

[0045] Figure 13 Module schematic diagram of a sample generation module in one embodiment of a training device for a target detection model according to the present disclosure;

[0046] Figure 14 Module schematic diagram of yet another embodiment of a training device for a target detection model according to the present disclosure;

[0047] Figure 15 Module schematic diagram of one embodiment of a target detection device according to the present disclosure;

[0048] Figure 16 Module schematic diagram of another embodiment of a target detection device according to the present disclosure. DETAILED DESCRIPTION

[0049] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which example embodiments of the present disclosure are described. The technical solutions in the embodiments of the present disclosure will be described clearly and completely in the present disclosure with reference to the accompanying drawings and embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure. The technical solutions of the present disclosure will be described in multiple aspects with reference to the accompanying drawings and embodiments.

[0050] In the following, “first”, “second”, etc. are only used to distinguish the above, and have no other special meanings.

[0051] In the related art known to the inventors, in the field of time action detection, a DETR model is used to detect action content in a video. The DERT model includes a backbone network, an encoder and a decoder based on a Transformer structure, i.e., a Transformer encoder and a Transformer decoder. An original video sequence is input into the Transformer encoder after being extracted by the backbone network (e.g., a convolutional neural network) to obtain a time and space feature map and position encoding information, and then the original video sequence is combined into an embedding vector. The Transformer encoder extracts image encoding features through a self-attention mechanism, and inputs the image encoding features and query features into the Transformer decoder. The Transformer decoder outputs a target query vector, and the target query vector is input into a classification head and a regression head constructed by a fully connected layer and a multi-layer perception layer to output the position and category of a detected target, which can be an action such as walking or running.

[0052] The Transformer structure has good performance in feature representation, and the performance of a video action detection method can be effectively improved by constructing a model using the Transformer. The Transformer encoder includes multiple encoder layers, and each encoder layer includes a multi-head self-attention layer, two layer normalization layers, and a feedforward neural network layer. The Transformer decoder includes multiple decoder layers, and each decoder layer includes two multi-head self-attention layers, three normalization layers, and a feedforward neural network layer.

[0053] The DETR method takes a fixed number N of learnable query features as input, each query feature is adaptively sampled from a two-dimensional image by the network, and information interaction between the query features is performed through self-attention. Finally, each query feature is used to predict the position and category of a detection box. In the field of time action detection, a fixed number of detection targets are predicted through an encoder-decoder manner. When detecting the targets, a Transformer structure based on sparse sampling is used to extract time segment features.

[0054] For the Transformer decoder, K trainable query features are used as input. Each query feature is a learnable vector that can extract temporal features from a specific moment based on learned statistical information. Self-attention is used to facilitate information exchange between all query features. Each query feature can be passed through a fully connected layer to predict the normalized coordinates of k sampled points across N time dimensions, and features are extracted from video features based on these sampled points to update the query features. For example, another fully connected layer is used to predict k weights from the input query features, and the k sampled features are summed with weights. The updated query features are then used by a regression head and a classification head to predict the location and type of the action. The regression head consists of a three-layer fully connected layer, and the classification head consists of a one-layer fully connected layer. The regression head predicts the normalized coordinates of the start and end of the action, while the classification head predicts the action's classification and confidence score.

[0055] In the existing DETR model, each video segment obtains the same number of positive samples as the number of labels through matching. The number of positive samples is less than the number of negative samples. Due to the insufficient number of positive samples participating in the training, the classification training of the DETR model is not sufficient, resulting in low classification accuracy.

[0056] Figure 1 This is a flowchart illustrating an embodiment of the training method for the object detection model according to the present disclosure. The object detection model includes an encoder module and a decoder module, etc. Figure 1 As shown:

[0057] Step 101: Obtain augmented samples corresponding to the video samples, and generate training samples based on the video samples and augmented samples.

[0058] In one embodiment, the video sample is a video clip with labeled information, and the augmented samples include positive and negative samples corresponding to the video sample. Training samples can be generated based on the video sample and the augmented samples using various existing methods. For example, if the video sample is a video clip containing running motion features, and another video also contains running motion features, then this video can be used as a positive sample. If another video contains features of other motions, such as long jump or singing, then this video can be used as a negative sample.

[0059] Step 102: Using the encoder module and based on the video feature information corresponding to the training samples, generate query feature information and construct a first loss function corresponding to the augmented samples.

[0060] In one embodiment, such as Figure 4As shown, the preset backbone network is used to process the training sample to generate video feature information, and the backbone network includes a neural network model, etc. The target detection model is a DETR model, and the encoder module includes a Transformer structure-based encoder, etc. The query feature is generated by the Transformer encoder, and the query feature can be a query vector, etc. The first loss function corresponding to the augmented sample is constructed.

[0061] In step 103, the decoder module is used to generate the first classification confidence information corresponding to the video sample and the regression information used to represent the target position, the second classification confidence information corresponding to the augmented sample based on the query feature information, and the second loss function corresponding to the training sample is constructed.

[0062] In one embodiment, the decoder module includes a Transformer structure-based decoder, and can also include a classification head, a regression head, etc. The decoder is a Transformer decoder, and the first classification confidence information, the regression information, and the second classification confidence information are generated by the Transformer decoder and the classification head, the regression head, etc., and the second loss function corresponding to the training sample is constructed.

[0063] In step 104, the first loss function and the second loss function are used to adjust the target detection model.

[0064] In one embodiment, the existing various model adjustment methods can be used to adjust the target detection model according to the first loss function and the second loss function, so that the function values of the first loss function and the second loss function are respectively within the allowed value range.

[0065] In one embodiment, various methods can be used to obtain the positive sample and the negative sample corresponding to the video sample. The first action type label of the video sample is determined, and based on the first action type label, the positive sample corresponding to the video sample is obtained from other videos, and the second action type label of the positive sample is the same as the first action type label. The other videos can be video clips labeled in the video library.

[0066] Based on the first action type label, the negative sample corresponding to the video sample is obtained from other videos, and the second action type label of the positive sample is different from the first action type label; or, the first video clip corresponding to the first action type label is determined, and the video clip is extracted from the video sample based on the first video clip as the negative sample, wherein the ratio of the overlap length to the length of the first video clip and the second video clip is less than the preset intersection-over-union threshold.

[0067] By adding the augmented samples corresponding to the video samples in the training, the feature difference of different action classes can be increased, and the discriminability of the features can be improved. For example, for the ground-truth action segment s i given in the video sample v g , the ground-truth action segment s g is manually labeled, and the labeling information includes the start and end time of the action in the ground-truth action segment, and the corresponding action type, and the corresponding label c g (the first action type label). Ground-truth action segments with the same label (the second action type label) as the label c g are sampled from other videos as positive samples of the ground-truth action segment s g .

[0068] There are two methods to obtain negative samples: (1) ground-truth action segments with different labels (the second action type label) from the label c g are sampled from other videos as negative samples; (2) random sampling is performed from the inside of the ground-truth action segment s g (the first video segment corresponding to the first action type label) to obtain a segment with an IoU less than a certain threshold as a negative sample. The IoU is the intersection over union of two segments, that is, IoU = the length of the overlap of the two segments / the sum of the lengths of the two segments. The closer the IoU is to 1, the more the two segments overlap.

[0069] In an embodiment, the first loss function corresponding to the augmented sample can be constructed in multiple ways. Figure 2 The flowchart for constructing the first loss function in one embodiment of the training method of the object detection model according to the present disclosure is shown in Figure 2 .

[0070] In step 201, segment feature information corresponding to the positive samples and the negative samples is obtained based on the video feature information.

[0071] In an embodiment, as shown in Figure 4 , the encoder module includes a fully connected layer and a region of interest pooling RoIPooling layer. The video feature information is processed through the fully connected layer to obtain fully connected features corresponding to the video feature information. The fully connected information is processed for feature extraction through the RoI Pooling layer to obtain the segment feature information.

[0072] The fully connected layer and the RoI Pooling layer can be existing fully connected layers and RoI Pooling layers. For example, Figure 5As shown, the Transformer decoder includes a self-attention module, a cross-attention module, two normalization layers, and a feedforward network. The Transformer decoder can use a variety of existing implementations.

[0073] Step 202: Generate the first loss function based on the fragment feature information.

[0074] There are several methods for generating the first loss function based on fragment feature information. Figure 3 This is a flowchart illustrating the generation of a first loss function based on fragment feature information in one embodiment of the training method for the object detection model according to this disclosure, as shown below. Figure 3 As shown:

[0075] Step 301: Obtain the first segment feature corresponding to the video sample and the second segment feature corresponding to the positive sample from the segment information.

[0076] Step 302: Determine the first sample features based on the first segment features and the second segment features.

[0077] Step 303: Obtain the third segment feature corresponding to the negative sample from the segment information.

[0078] Step 304: Determine the second sample features based on the first segment features, the second segment features, and the third segment features.

[0079] Step 305: Generate a first loss function based on the features of the first sample and the features of the second sample.

[0080] In one embodiment, based on such Figure 4 The network structure shown obtains x∈R for video segment s. T×D′ and These represent the video features extracted by the pre-trained network and the features further projected through a single fully connected layer, respectively. Here, x represents the features extracted from a video segment using a pre-trained network, and T indicates that the video segment has T frames. The feature dimension of each frame is D'. By projecting each frame (using a fully connected layer), the features of each frame can be projected into D dimensions. The RoI Pooling layer is used to extract features within a specific time segment, and the average value is calculated along the time dimension (making the time dimension 1). This average value is then used as the feature of the video segment s, thus extracting the segment's feature information.

[0081] Contrastive learning is used to enhance the features of the input decoder, resulting in a loss function ACE-enc. The first loss function is generated as follows:

[0082]

[0083] wherein f is a segment feature of a certain video segment obtained through a full connection and a RoI Pooling layer, i.e., a first segment feature corresponding to the video sample s; f p is a segment feature of a positive sample obtained by using the same method, i.e., a second segment feature corresponding to the positive sample; f T f p is a first sample feature; D is a set composed of the positive sample and the negative sample, f j is a second segment feature corresponding to the positive sample or a third segment feature corresponding to the negative sample; f T f j is a second sample feature.

[0084] In an embodiment, the construction of the second loss function corresponding to the training sample can use multiple methods. Figure 6 FIG. 2 is a flowchart of the construction of the second loss function in an embodiment of the training method of the object detection model according to the present disclosure, as shown in Figure 6 .

[0085] Step 601, determining a first classification loss function corresponding to the first classification confidence information.

[0086] Step 602, determining a second classification loss function corresponding to the second classification confidence information.

[0087] Step 603, generating a second loss function based on the second classification loss function and the second classification loss function.

[0088] In an embodiment, the first classification loss function can be a first cross-entropy loss function, and the second classification loss function can be a second cross-entropy loss function. The first cross-entropy loss function and the second cross-entropy loss function can be multiple existing cross-entropy loss functions. The sum of the first cross-entropy loss function and the second cross-entropy loss function is taken as the second loss function.

[0089] For example, as shown in Figure 7 , the classification performance is improved by increasing the training samples of the classification head. Since the number of positive samples is increased, the loss of ACE-dec is generated. In order to increase the training amount, the training samples containing the video samples and the enhanced samples are used in the training stage. The newly added enhanced samples output features through the backbone network for extracting visual features, the encoder module for further encoding the video features, and the decoder network for detecting actions. The features are sampled to additionally train the classification head (classifier) to train the classifier. At the same time, the features of the video samples are used to train the classification head and the regression head. By using the additional enhanced sample segments, the number of positive samples can be increased, thereby improving the training performance.

[0090] According to the ACE-dec loss of each layer of the Transformer decoder, a second loss function is constructed:

[0091]

[0092] wherein, is a first cross-entropy loss function of the first classification confidence information corresponding to the video sample, representing the classification loss of each video sample; is a second cross-entropy loss function of the second classification confidence information corresponding to the enhanced sample, representing the classification loss of the matched enhanced sample; y represents a label, only when the query feature y (not empty) is matched successfully will the corresponding action segment be generated, thereby generating a loss, that is, only the query position of the matched enhanced sample is added with an additional loss.

[0093] and can be implemented using existing methods, for example and are both existing SigmoidFocal Loss, and Focal loss is an existing cross-entropy loss function. is a loss function corresponding to the predicted segment of the video sample, is a loss function corresponding to the predicted segment according to the labeled segment of the enhanced sample.

[0094] The training method of the object detection model of the present disclosure generates training samples based on video samples and enhanced samples, thereby enhancing the distinguishability of input video features; by generating classification confidence information corresponding to the video sample and the enhanced sample respectively, the classification performance in the field of video action detection is improved.

[0095] Figure 8 is a flowchart of one embodiment of the object detection method according to the present disclosure, as shown in Figure 8 .

[0096] Step 801, obtaining a trained object detection model; wherein the object detection model is trained by the training method as above.

[0097] Step 802, using the object detection model and based on a video to be detected, generating classification confidence information corresponding to the video to be detected and regression information for representing target positions.

[0098] In one embodiment, the trained target detection model is input with a video to be detected, and the target detection model outputs classification confidence, regression information for representing a target position; the target is an action in the video to be detected, and the classification confidence information can be a score of action classification confidence, and the regression information can be start and end information of the action.

[0099] In one embodiment, the present disclosure provides a training device 90 of a target detection model, which includes an encoder module and a decoder module. The training device 90 includes a sample generation module 91, an encoding processing module 92, a decoding processing module 93, and a model adjustment module 94. The sample generation module 91 obtains an enhanced sample corresponding to a video sample, and generates a training sample based on the video sample and the enhanced sample.

[0100] The encoding processing module 92 uses the encoder module and generates query feature information based on video feature information corresponding to the training sample, and constructs a first loss function corresponding to the enhanced sample. The decoding processing module 93 uses the decoder module and generates first classification confidence information corresponding to the video sample and regression information for representing a target position based on the query feature information, second classification confidence information corresponding to the enhanced sample, and constructs a second loss function corresponding to the training sample. The model adjustment module 94 adjusts the target detection model using the first loss function and the second loss function.

[0101] As shown in Figure 10 , the training device 90 of the target detection model further includes a feature information acquisition module 95, which uses a preset backbone network to process the training sample to generate video feature information; wherein the backbone network includes a neural network model.

[0102] In one embodiment, the enhanced sample includes a positive sample and a negative sample corresponding to the video sample. As shown in Figure 11 , the encoding processing module 92 includes a segment feature acquisition unit 921 and a first loss determination unit 922. The segment feature acquisition unit 921 acquires segment feature information corresponding to the positive sample and the negative sample based on the video feature information. The first loss determination unit 922 generates the first loss function according to the segment feature information.

[0103] The encoder module includes a fully connected layer and a region of interest pooling RoI Pooling layer. The segment feature acquisition unit 921 processes the video feature information through the fully connected layer to obtain fully connected features corresponding to the video feature information; and the segment feature acquisition unit 921 performs feature extraction processing on the fully connected information through the RoI Pooling layer to obtain the segment feature information.

[0104] The first loss determination unit 922 obtains the first segment feature corresponding to the video sample and the second segment feature corresponding to the positive sample from the segment information; the first loss determination unit 922 determines the first sample feature based on the first segment feature and the second segment feature.

[0105] The first loss determination unit 922 obtains the third segment feature corresponding to the negative sample from the segment information, and determines the second sample feature based on the first segment feature, the second segment feature and the third segment feature; the first loss determination unit 922 generates a first loss function based on the first sample feature and the second sample feature.

[0106] In one embodiment, such as Figure 12 As shown, the decoding processing module 93 includes a classification loss determination unit 931 and a second loss determination unit 932. The classification loss determination unit 931 determines a first classification loss function corresponding to the first classification confidence information and a second classification loss function corresponding to the second classification confidence information. The second loss determination unit 932 generates a second loss function based on the second classification loss function and the second classification loss function.

[0107] The first classification loss function includes a first cross-entropy loss function; the second classification loss function includes a second cross-entropy loss function. The second loss determination unit 932 uses the sum of the first cross-entropy loss function and the second cross-entropy loss function as the second loss function.

[0108] In one embodiment, such as Figure 13 As shown, the sample generation module 91 includes a positive sample acquisition unit 911 and a negative sample acquisition unit 912. The positive sample acquisition unit 911 determines the first action type label of the video sample; based on the first action type label, the positive sample acquisition unit 911 acquires positive samples corresponding to the video sample from other videos; wherein, the second action type label of the positive sample is the same as the first action type label.

[0109] The negative sample acquisition unit 912 acquires negative samples corresponding to the video samples from other videos based on the first action type label; wherein, the second action type label of the positive sample is different from the first action type label.

[0110] The negative sample acquisition unit 912 determines the first video segment corresponding to the first action type label, and extracts video segments from the video samples based on the first video segment as negative samples; wherein, the ratio of the overlap length of the first video segment and the sum of the lengths of the second video segment is less than a preset intersection-union ratio threshold.

[0111] In one embodiment, such as Figure 14As shown, this disclosure provides a training apparatus for an object detection model, which may include a memory 141, a processor 142, a communication interface 143, and a bus 144. The memory 141 is used to store instructions, and the processor 142 is coupled to the memory 141. The processor 142 is configured to execute the above-described training method for the object detection model based on the instructions stored in the memory 141.

[0112] The memory 141 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 141 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 142 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the training method of the object detection model disclosed herein.

[0113] In one embodiment, this disclosure provides a target detection device 15, including a model acquisition module 151 and a detection processing module 152. The model acquisition module 151 acquires a trained target detection model; wherein, the target detection model is trained using the training method described above; the detection processing module 152 uses the target detection model and, based on the video to be detected, generates classification confidence information corresponding to the video to be detected and regression information for characterizing the target location.

[0114] In one embodiment, such as Figure 16 As shown, this disclosure provides a target detection device that may include a memory 161, a processor 162, a communication interface 163, and a bus 164. The memory 161 is used to store instructions, and the processor 162 is coupled to the memory 161. The processor 162 is configured to execute the target detection method described above based on the instructions stored in the memory 161.

[0115] The memory 161 can be a high-speed RAM, non-volatile memory, or a memory array. The memory 161 may also be divided into blocks, and these blocks can be combined into virtual volumes according to certain rules. The processor 162 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the target detection method of this disclosure.

[0116] In one embodiment, the present disclosure provides a computer readable storage medium storing computer instructions, which, when executed by a processor, implement the method in any one of the above embodiments.

[0117] The training method of the target detection model, the target detection method, the device and the storage medium in the above embodiments can generate training samples based on video samples and enhanced samples and construct a loss function corresponding to the enhanced samples, can enhance the distinguishability of the input video features, improve the feature similarity of the same action, can increase the feature difference of different actions and improve the distinguishability of the features, generate classification confidence information corresponding to the video samples and the enhanced samples based on the query feature information and construct a loss function, increase the number of positive samples and other training samples, fully train the classification of the model, improve the classification performance and the training performance, and make the prediction result more accurate, thereby improving the user's use experience.

[0118] The methods and systems of the present disclosure can be implemented in many ways. For example, the methods and systems of the present disclosure can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above described order of steps for the methods is merely for illustration, and the steps of the methods of the present disclosure are not limited to the above specifically described order, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure can also be implemented as programs recorded in recording media, which include machine readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers the recording media storing the programs for executing the methods according to the present disclosure.

[0119] The description of the present disclosure is given for the purpose of illustration and description, and is not exhaustive or limiting to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are chosen and described in order to best explain the principles of the present disclosure and its practical application, and to enable others skilled in the art to understand the present disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A training method for an object detection model, wherein, The object detection model includes an encoder module and a decoder module; the training method includes: Obtain augmented samples corresponding to video samples, and generate training samples based on the video samples and the augmented samples; Using the encoder module and based on the video feature information corresponding to the training samples, query feature information is generated, and a first loss function corresponding to the augmented samples is constructed. Using the decoder module and based on the query feature information, a first classification confidence information and regression information for characterizing the target location corresponding to the video sample are generated, as well as a second classification confidence information corresponding to the augmented sample, and a second loss function corresponding to the training sample is constructed. The construction of the second loss function corresponding to the training samples includes: Determine a first classification loss function corresponding to the first classification confidence information; determine a second classification loss function corresponding to the second classification confidence information; generate the second loss function based on the second classification loss function and the second classification loss function. The target detection model is adjusted using the first loss function and the second loss function.

2. The method as described in claim 1, wherein, The augmented samples include: positive samples and negative samples corresponding to the video samples; the construction of the first loss function corresponding to the augmented samples includes: Based on the video feature information, segment feature information corresponding to the positive and negative samples is obtained; A first loss function is generated based on the fragment feature information.

3. The method as described in claim 2, wherein, The encoder module includes: a fully connected layer and a region of interest pooling (RoIPooling) layer; the step of obtaining segment feature information corresponding to the positive and negative samples based on the video feature information includes: The video feature information is processed through the fully connected layer to obtain the fully connected features corresponding to the video feature information; The RoIPooling layer is used to extract features from the fully connected features to obtain the fragment feature information.

4. The method as described in claim 2, wherein generating the first loss function based on the fragment feature information comprises: From the segment feature information, obtain the first segment feature corresponding to the video sample and the second segment feature corresponding to the positive sample; Based on the features of the first segment and the features of the second segment, the features of the first sample are determined; Obtain the third segment feature corresponding to the negative sample from the segment feature information; Based on the first segment features, the second segment features, and the third segment features, the second sample features are determined; The first loss function is generated based on the first sample features and the second sample features.

5. The method of claim 1, wherein the first classification loss function comprises: First cross-entropy loss function; The second classification loss function includes: a second cross-entropy loss function; generating the second loss function based on the second classification loss function includes: The sum of the first cross-entropy loss function and the second cross-entropy loss function is used as the second loss function.

6. The method of claim 2, further comprising: Determine the first action type label of the video sample; Based on the first action type label, the positive sample corresponding to the video sample is obtained from other videos; wherein the second action type label of the positive sample is the same as the first action type label.

7. The method of claim 6, comprising: Based on the first action type label, the negative sample corresponding to the video sample is obtained from other videos; wherein the second action type label of the positive sample is different from the first action type label.

8. The method of claim 6, comprising: Determine the first video segment corresponding to the first action type label; A second video segment is extracted from the video sample based on the first video segment, and used as the negative sample; wherein the ratio of the overlap length of the first video segment and the second video segment to the sum of their lengths is less than a preset intersection-union ratio threshold.

9. The method of claim 1, comprising: The training samples are processed using a preset backbone network to generate the video feature information; wherein, the backbone network includes a neural network model.

10. The method according to any one of claims 1 to 9, wherein, The encoder module includes: an encoder based on the Transformer architecture; The decoder module includes a decoder based on the Transformer architecture.

11. A target detection method, comprising: A pre-trained object detection model is obtained; wherein the object detection model is trained using the training method described in any one of claims 1 to 10. Using the target detection model and based on the video to be detected, classification confidence information corresponding to the video to be detected and regression information for characterizing the target location are generated.

12. A training device for an object detection model, wherein, The object detection model includes an encoder module and a decoder module; the training device includes: The sample generation module is used to obtain augmented samples corresponding to video samples, and generate training samples based on the video samples and the augmented samples. The encoding processing module is used to generate query feature information using the encoder module and based on video feature information corresponding to the training samples, and to construct a first loss function corresponding to the augmented samples; The decoding processing module is used to generate, using the decoder module and based on the query feature information, first classification confidence information corresponding to the video sample and regression information for characterizing the target location, second classification confidence information corresponding to the enhanced sample, and constructing a second loss function corresponding to the training sample; Specifically, the decoding processing module is used to determine a first classification loss function corresponding to the first classification confidence information; determine a second classification loss function corresponding to the second classification confidence information; and generate the second loss function based on the second classification loss function and the second classification loss function. The model adjustment module is used to adjust the target detection model using the first loss function and the second loss function.

13. A training device for an object detection model, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 10 based on instructions stored in the memory.

14. A target detection device, comprising: A model acquisition module is used to acquire a trained object detection model; wherein the object detection model is trained by the training method described in any one of claims 1 to 10; The detection processing module is used to generate classification confidence information corresponding to the video to be detected and regression information for characterizing the target location, based on the target detection model and the video to be detected.

15. A target detection device, comprising: Memory; and a processor coupled to the memory, the processor being configured to perform the method of claim 11 based on instructions stored in the memory.

16. A computer-readable storage medium that non-transitoryly stores computer instructions, which are executed by a processor according to any one of claims 1 to 11.

Images