A training method and related device for target detection model

Abstract

The present application discloses a method for training a target detection model, including: using the backbone network of the model to be trained to perform feature extraction operations on an image to obtain a multi-scale feature map; encoding the multi-scale feature map based on a multi-scale deformable attention encoding module to obtain encoded image features; performing sequence construction processing based on the correct annotation of the image to obtain a target sequence; decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence; and updating the parameters of the model to be trained based on a preset loss function, the predicted sequence, and the correctly annotated target sequence of the image. This makes it possible to process feature maps with high complexity and improve the efficiency and performance of model processing. The present application also discloses a training device, a server, and a computer-readable storage medium for a target detection model, which have the above beneficial effects.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a training method, training device, server and computer-readable storage medium for a target detection model. Background Art

[0002] In the related technology, although PIX2SEQ (a target detection architecture) converts the target detection task into the generation task of the language model, it still uses the transformer internally. Due to the characteristics of the transformer itself, when processing image features, it will calculate the relationship between the current position and all other positions. The complexity is proportional to the size of the feature map, which makes it impossible to use large-size or multi-size features, thereby affecting the performance of the model.

[0003] Therefore, how to maintain the efficiency during the treatment process is a key issue that technicians in this field are concerned about. Summary of the invention

[0004] The purpose of this application is to provide a training method, training device, server and computer-readable storage medium for a target detection model, so that high-complexity feature maps can be processed and the efficiency and performance of model processing can be improved.

[0005] In order to solve the above technical problems, the present application provides a method for training a target detection model, comprising:

[0006] Use the backbone network of the model to be trained to perform feature extraction on the image to obtain a multi-scale feature map;

[0007] Encoding the multi-scale feature map based on a multi-scale deformable attention encoding module to obtain encoded image features;

[0008] Performing sequence construction processing based on the correct annotation of the image to obtain a target sequence;

[0009] Decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence;

[0010] The parameters of the model to be trained are updated based on a preset loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0011] Optionally, a backbone network of the model to be trained is used to perform feature extraction operations on the image to obtain a multi-scale feature map, including:

[0012] Performing image enhancement processing on the image to obtain a transformed image;

[0013] An image feature extraction operation is performed on the transformed image based on the backbone network to obtain the multi-scale feature map.

[0014] Optionally, encoding the multi-scale feature map based on a multi-scale deformable attention encoding module to obtain encoded image features includes:

[0015] Acquire image features of each scale from the multi-scale feature map;

[0016] Encoding the position of each feature point in the image feature to obtain a position code;

[0017] Fusion of the fusion features of the image features at each scale with the position code to obtain a code input;

[0018] The encoding input is encoded based on the multi-scale deformable attention encoding module to obtain the encoded image features.

[0019] Optionally, performing sequence construction processing based on the correct annotation of the image to obtain a target sequence includes:

[0020] Normalizing the size of the correct marks in the image to obtain a normalized image;

[0021] A sequence construction process is performed on any target in the normalized image to obtain the target sequence.

[0022] Optionally, updating parameters of the model to be trained based on a preset loss function, the prediction sequence, and a correctly labeled target sequence of the image includes:

[0023] Constructing a loss function based on the model structure of the model to be trained;

[0024] Parameters of the model to be trained are updated based on the loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0025] Optionally, also include:

[0026] When the training of the model to be trained is completed, the model to be trained is used as the target detection model.

[0027] Optionally, also include:

[0028] Testing the target detection model based on the labeled data;

[0029] When the test passes, a test success message is sent.

[0030] The present application also provides a training device for a target detection model, comprising:

[0031] A feature extraction module is used to perform feature extraction operations on the image using the backbone network of the model to be trained to obtain a multi-scale feature map;

[0032] A feature encoding module, used for encoding the multi-scale feature map based on the multi-scale deformable attention encoding module to obtain encoded image features;

[0033] A sequence construction module, used for performing sequence construction processing based on the correct annotation of the image to obtain a target sequence;

[0034] A feature decoding module, used for decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence;

[0035] A parameter updating module is used to update the parameters of the model to be trained based on a preset loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0036] The present application also provides a server, comprising:

[0037] Memory for storing computer programs;

[0038] A processor is used to implement the steps of the training method of the target detection model as described above when executing the computer program.

[0039] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the training method of the target detection model as described above are implemented.

[0040] The present application provides a method for training a target detection model, comprising: using a backbone network of a model to be trained to perform a feature extraction operation on an image to obtain a multi-scale feature map; encoding the multi-scale feature map based on a multi-scale deformable attention encoding module to obtain encoded image features; performing sequence construction processing based on the correct annotation of the image to obtain a target sequence; decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence; and updating parameters of the model to be trained based on a preset loss function, the predicted sequence, and the correctly annotated target sequence of the image.

[0041] By introducing the deformation attention mechanism into the encoder and decoder of the transformer successively, the complexity of the transformer in processing image features is greatly reduced, so that the model can use high-resolution image features and multi-scale feature information to further improve the model performance.

[0042] The present application also provides a training device, a server and a computer-readable storage medium for a target detection model, which have the above beneficial effects and are not elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A flowchart of a method for training a target detection model provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the structure of a method for training a target detection model provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the structure of a training device for a target detection model provided in an embodiment of the present application;

[0047] Figure 4 A structural schematic diagram of a feature graph processing method for a target detection model training method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The core of this application is to provide a training method, training device, server and computer-readable storage medium for a target detection model, so that high-complexity feature maps can be processed and the efficiency and performance of model processing can be improved.

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] In related technologies, target detection is considered as a mapping from an input image to a set. For example, DETR (DEtectionTransformer, end-to-end target detection based on Transformers) and Deformable-DETR (DETR based on attention mechanism) are two algorithms that introduce transformer technology from NLP (Natural Language Processing) into target detection. Target detection is considered as a prediction of a set, which is truly end-to-end training. First, the ResNet backbone network extracts the features of the entire image, then the feature map is added with position information and sent to the transformer, and finally the FFN (feed forward network) outputs the detection results. In DETR, since the transformer calculates the relationship between the current query (feature point) and all keys during the processing, the number of keys is usually very large, and the processing complexity is proportional to the size of the feature map. An average strategy is adopted for all keys during initialization, and the attention mechanism ultimately only focuses on a small number of keys, resulting in slow convergence. It also results in only small-scale features being able to be used, and small-scale features affect the performance of small target detection. To address this problem, in an embodiment of the present application, the idea of ​​deformable convolution is introduced into the transformer, and the transformer attention is replaced by multi-scale deformable attention, which reduces the focus of the entire image to points around the reference point, greatly reducing the computational complexity, accelerating the convergence speed, and improving the detection performance, especially for the detection of small targets.

[0051] The target detection is considered as the description of the image, that is, the detection box and the corresponding category are considered as the description of the image. The model receives the image input and generates the expected description sequence. The processing flow first uses the backbone network to extract the features of the enhanced image, and constructs a sequence of bbox (rectangular box position information) and corresponding categories; then the transformer encoder (encoding module) encodes the image features; finally, through the generative transformer decoder (decoding module), based on the previous sequence and the encoded image representation, the expected sequence, that is, the detection result, is output.

[0052] Although PIX2SEQ (an object detection framework) converts the object detection task into a language model generation task, it still uses transformer internally. Due to the characteristics of the transformer itself, when processing image features, it calculates the relationship between the current position and all other positions. The complexity is proportional to the size of the feature map, which makes it impossible to use large-size or multi-size features, which in turn affects the performance of the model.

[0053] Therefore, the present application provides a training method for a target detection model, which greatly reduces the complexity of the transformer in processing image features by successively introducing the deformation attention mechanism into the encoder and decoder of the transformer, so that the model can use high-resolution image features and multi-scale feature information to further improve the model performance.

[0054] The following is an example of an example of a method for training a target detection model provided by the present application.

[0055] Please refer to Figure 1 , Figure 1 A flowchart of a method for training a target detection model provided in an embodiment of the present application.

[0056] In this embodiment, the method may include:

[0057] S101, using the backbone network of the model to be trained to perform feature extraction operations on the image to obtain a multi-scale feature map;

[0058] It can be seen that this step aims to extract the corresponding features from the image first. The backbone network can adopt any main network structure of the target detection network provided by the prior art, and is not specifically limited here.

[0059] Furthermore, this step may include:

[0060] Step 1, performing image enhancement processing on the image to obtain a transformed image;

[0061] Step 2: Perform image feature extraction on the transformed image based on the backbone network to obtain a multi-scale feature map.

[0062] It can be seen that this step mainly explains how to extract feature maps. In this optional solution, the image is enhanced to obtain a transformed image, and the image feature extraction operation is performed on the transformed image based on the backbone network to obtain a multi-scale feature map. Among them, the image enhancement processing includes random geometric transformation and random color transformation, among which the random geometric transformation includes random shearing, random horizontal flipping, random resizing, etc., and the random color transformation includes color jitter, random adjustment of image contrast, brightness, saturation, etc.

[0063] S102, encoding the multi-scale feature map based on the multi-scale deformable attention encoding module to obtain encoded image features;

[0064] Based on S101, this step aims to encode the multi-scale feature map based on the multi-scale deformable attention encoding module to obtain the encoded image features.

[0065] That is, based on the original attention encoding, multi-scale deformable attention encoding is realized.

[0066] Please refer to Figure 4 , Figure 4 A structural schematic diagram of a feature graph processing method for a target detection model training method provided in an embodiment of the present application.

[0067] Among them, the encoded image features can be used as follows Figure 4 The feature map processing method shown here is not specifically limited. The processed feature map then enters the multi-scale deformable attention module for further processing. Under normal circumstances, multi-scale information can effectively improve the target detection performance, but the high-level feature map has an over-smoothing problem due to the convolution operation. To avoid this problem, multi-scale information is fully utilized at the same time, and the corresponding feature points of each layer of feature map are obtained according to the layer level, and fused through the multi-scale attention module.

[0068] Furthermore, this step may include:

[0069] Step 1, obtaining image features of each scale from the multi-scale feature map;

[0070] Step 2, encoding the position of each feature point in the image feature to obtain a position code;

[0071] Step 3, fuse the fusion features of the image features at each scale with the position code to obtain the code input;

[0072] Step 4: Encode the encoded input based on the multi-scale deformable attention encoding module to obtain the encoded image features.

[0073] It can be seen that this optional solution mainly explains how to perform feature encoding. In this optional solution, image features of each scale are obtained from the multi-scale feature map, the position of each feature point in the image feature is encoded to obtain the position encoding, the fusion feature of the image feature of each scale is fused with the position encoding to obtain the encoding input, and the encoding input is encoded based on the multi-scale deformable attention encoding module to obtain the encoded image feature.

[0074] S103, performing sequence construction processing based on the correct annotation of the image to obtain a target sequence;

[0075] Based on S101, this step aims to perform sequence construction processing based on the correct annotation of the image to obtain the target sequence.

[0076] That is, generate the sequence content that the decoding process needs to adopt.

[0077] Furthermore, this step may include:

[0078] Step 1, normalize the size of the correct mark in the image to obtain a normalized image;

[0079] Step 2: Perform sequence construction processing on all targets in the normalized image to obtain a target sequence.

[0080] It can be seen that this optional solution mainly explains how to obtain the target sequence. In this optional solution, the correct mark in the image is normalized to obtain a normalized image, and a sequence is constructed for any target in the normalized image to obtain the target sequence.

[0081] S104, decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence;

[0082] Based on S102 and S103, this step aims to decode the encoded image features and the target sequence based on the multi-scale deformable attention decoding module to obtain a predicted sequence. That is, to implement the corresponding decoding operation. What is different from the prior art is that the multi-scale deformable attention decoding module is used in this step, so that it can focus on important features and improve the performance of processing complex features.

[0083] S105, updating the parameters of the model to be trained based on the preset loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0084] On the basis of S104, this step aims to update the parameters of the to-be-trained model based on the preset loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0085] Furthermore, this step may include:

[0086] Step 1, construct a loss function based on the model structure of the model to be trained;

[0087] Step 2: Update the parameters of the model to be trained based on the loss function, the prediction sequence, and the correctly labeled target sequence of the image.

[0088] It can be seen that this optional solution mainly describes how to update parameters. In this optional solution, a loss function is constructed based on the model structure of the model to be trained, and the parameters of the model to be trained are updated based on the loss function, the prediction sequence and the correctly labeled target sequence of the image.

[0089] Furthermore, this embodiment may also include:

[0090] When the training of the model to be trained is completed, the model to be trained is used as the target detection model.

[0091] It can be seen that in this optional solution, when the training is completed, the corresponding target detection model is obtained to improve the accuracy of the target detection model.

[0092] Furthermore, this embodiment may also include:

[0093] Step 1: Test the target detection model based on the labeled data;

[0094] Step 2: When the test passes, a test success message is sent.

[0095] It can be seen that this optional solution mainly describes testing the target detection model. In this optional solution, the target detection model is tested based on the labeled data, and when the test passes, a test success message is sent.

[0096] In summary, this embodiment greatly reduces the complexity of the transformer in processing image features by introducing the deformation attention mechanism into the encoder and decoder of the transformer, so that the model can use high-resolution image features and multi-scale feature information to further improve the model performance.

[0097] The following is a specific example to further illustrate the training method of a target detection model provided by the present application.

[0098] In this embodiment, the following four parts are mainly included:

[0099] The backbone network extracts image features and constructs multi-scale features; constructs a Transformer encoder based on a multi-scale deformable attention module; constructs a sequence based on the image's ground truth (correctly labeled data) and performs corresponding data enhancement; and constructs a generative Transformer decoder based on a multi-scale deformable attention module.

[0100] Among them, the backbone network is used to extract image features and construct multi-scale features. Object detection belongs to transfer learning, which is operated on the backbone and head of the model. The backbone is generally obtained through pre-training of other data sets. The head is another network used to bridge the backbone and object detection data. Therefore, the backbone directly affects the final object detection results. In addition, since each scale has its own focus area, if only the last layer of the backbone is used, the performance of small object detection will be poor. Therefore, information of different scales is usually considered comprehensively during object detection.

[0101] Among them, the Transformer encoder is constructed based on the multi-scale deformable attention module. The Multi-headattention mechanism allows the neural network to focus on more relevant elements. Since all keys (attention values) use the average value strategy at the beginning, and the number of keys is very large, it takes longer to focus on the relevant keys. In addition, the computational complexity and spatial complexity required by the Multi-head attention mechanism are relatively high, which makes it impossible to use high-resolution feature maps. Based on this, the deformable attention mechanism is introduced into the Transformer. By selecting a small number of keys for Multi-head attention calculations, the computational complexity and spatial complexity are greatly reduced. Subsequently, this operation is used in multi-scale feature maps, which further solves the problem that the Multi-head attention mechanism cannot use high-resolution feature maps.

[0102] Among them, the ground truth based on the real image is used to build the sequence, and the corresponding data enhancement is performed. The category in the target detection annotation can be directly represented by a token, but the detection box cannot. It needs to be quantized through a quantization strategy. After quantization, the detection box will be unified in the interval, so that it can be represented by a very small vocabulary. After quantizing each target in the image, the detection box and the category are randomly formed into a set of sequences. In order to improve the performance of the model, noise is added after the real sequence through the enhancement method to form a new sequence.

[0103] A generative Transformer decoder is constructed based on a multi-scale deformable attention module. Similar to the Transformer encoder, in order to reduce computational complexity and space complexity, a deformable attention mechanism is introduced in the cross-attention. Based on the image features and sequence after encoder encoding, the desired sequence is generated in a generative manner, and the maximum likelihood loss is constructed with the real ground truth. The model parameters are updated based on this loss.

[0104] Please refer to Figure 2 , Figure 2 A structural diagram of a method for training a target detection model provided in an embodiment of the present application.

[0105] See also Figure 2 , which is a flow chart of the end-to-end target detection model based on Transformer in the present invention. It is divided into the following six steps:

[0106] Step 1, extract input image features using the backbone network;

[0107] Step 2: Build a Transformer encoder based on the multi-scale deformable attention module;

[0108] Step 3: construct a sequence based on the ground truth of the image and perform corresponding data enhancement;

[0109] Step 4: Build a Transformer decoder based on the multi-scale deformable attention module;

[0110] Step 5, construct the loss function;

[0111] Step 6: Train the model and test it.

[0112] The above steps are described in detail below.

[0113] The backbone network is used to extract input image features. In order to improve the model detection performance, the image is first enhanced using image enhancement algorithms, and then a backbone network with richer representation is used to extract features, and multiple scales are constructed at the same time. This can include:

[0114] Step 1: Perform image enhancement on the input image, which is divided into random geometric transformation and random color transformation. The random geometric transformation includes random shearing, random horizontal flipping, random resizing, etc. The random color transformation includes color jitter, random adjustment of image contrast, brightness, saturation, etc.;

[0115] Step 2: Use a backbone network to extract image features from the transformed image, such as ResNeXt101.

[0116] Step 3: Construct multi-scale image features based on the backbone network.

[0117] Among them, the Transformer encoder is constructed based on the multi-scale deformable attention module. Although the Multi-head attention mechanism in Transformer has good performance in NLP, when it is introduced into CV (computer vision), the object to be processed is an image. When calculating the relationship between the current query (feature point) and all keys, the number of keys is very large, resulting in very high computational complexity and spatial complexity, forcing the model to use low-resolution image features. In order to solve this problem, a deformable attention mechanism is introduced to adaptively select a small number of keys at multiple scales, reduce complexity and fuse multi-scale information at the same time, thereby improving model performance.

[0118] Among them, the image feature position is first encoded and fused with the image features. The operation process is as follows:

[0119] Step 1: Use the backbone network to extract image features and construct multi-scale image features;

[0120] Step 2: Obtain image features at each scale, encode the position of each feature point, and then fuse the image features with the corresponding position encoding;

[0121] Step 3: Expand the features with positions into a row as input to the encoder.

[0122] Then, we build an encoder with a multi-scale deformable attention mechanism. In the image feature map, the transformer attention mechanism tries to check all positions in the space as much as possible. By introducing a deformable attention module, it only focuses on a small number of keys around the reference point, thus solving the spatial resolution and convergence problems of the feature map. Based on this, we build the transformer encoder module. The specific steps are as follows:

[0123] Step 1: Construct a single-scale deformable attention module. Let the input feature map be x∈R C×H×W , the feature point and the reference point are z q 、r q , where q is the corresponding index, and the deformable attention feature is calculated as follows:

[0124]

[0125] in, A hqk ∈[0,1], h is the index of the attention head, k is the index of the selected keys, K represents the number of all selected keys, Δr hqk represents the offset of the kth sample in the hth head, A hqk represents the weight corresponding to the kth sample in the hth head, Δr hqk is a floating point number, then r q +Δr hqk Usually a non-integer, the value corresponding to this position can be obtained based on the surrounding points through bilinear interpolation method.

[0126] Step 2: Construct an adaptive multi-scale deformable attention module. Generally, using multiple scale information can effectively improve the target detection performance, but the high-level feature map is processed by convolution and other processes on the shallow feature map. In this way, all point information of the shallow feature may be integrated into one or several points in the high-level feature map, resulting in smoothing problems. Therefore, in order to make full use of multiple scale information, a multi-scale deformable attention module is proposed based on the single-scale deformable attention module. Generally, multi-scale feature maps can improve the performance of target detection. Therefore, multi-scale is added based on the single-scale deformable attention module. The calculation method is as follows:

[0127]

[0128] in, A hlqk ∈[0,1], represents the input multi-scale feature map, h is the attention head index, l is the current feature layer, L is the total feature layer, k is the index of the selected keys, 2 (L-l) K represents the number of all selected keys required by the current layer, 2 (L-l) is the coefficient of K, which can ensure that low-level features select more keys, while high-level features select a small number of keys, Δr hlqk A represents the offset of the kth sample of the lth layer feature in the hth head, hlqk represents the weight corresponding to the kth sample of the lth layer feature in the hth head, Represents the coordinates after renormalizing the l-th layer features.

[0129] Step 3: Build a transformer encoder based on the multi-scale deformable attention module. Replace the attention module in the transformer with the multi-scale deformable attention module, set the number of heads H to 8, the number of sampling point keys K to 4, and unify the number of channels of multi-scale features to 256.

[0130] The sequence is constructed based on the ground truth of the image, and the corresponding data enhancement is performed. The ground truth of target detection has a detection box and a corresponding category. Due to different image sizes, the detection boxes are also different, so the images need to be unified to a unified size, so that all detection boxes can be quantized to a unified scale. A small vocabulary can be used to represent them, and then the token corresponding to the category is added. Based on this, all targets are constructed into a sequence, and the noise sequence is added using the data enhancement algorithm during training. The specific implementation method is as follows:

[0131] Step 1: Normalize the ground truth in the image to a uniform size.

[0132] Modify the original image according to the aspect ratio of the original image and the set longest side size, such as setting the longest side to 1400; adjust the bounding box size in the ground truth accordingly based on the image resize ratio, so that all images corresponding to the bounding box are normalized (quantized) to between [1, 1400]; randomly select any target in the image and construct a sequence. The bounding box in the target is normalized according to the above method, and the category is not modified; sample and transform the remaining targets in the same way and append them to the previous sequence.

[0133] Step 2: Data enhancement. That is, based on the generated sequence, randomly generate some sequences and append them to it, such as randomly adjusting the position of the upper left corner or lower right corner of the target box, translating the bounding box to other positions or modifying the corresponding category, etc.

[0134] Step 3: Use natural language processing to uniformly convert the generated sequences into corresponding tokens.

[0135] The Transformer decoder is constructed based on the multi-scale deformable attention module. The decoder is mainly composed of two modules: cross-attention and self-attention. In the cross-attention module, the query extraction features come from the feature map, where the key is the feature map output by the encoder; in the self-attention module, the queries interact with each other, and the key is related to the input query. Since the deformable attention module targets the feature map, only the cross-attention module is replaced by the multi-scale deformable attention module, and the self-attention module is the same as before.

[0136] Construct the loss function. Since the target detection task is converted into a language model generation task, the maximum likelihood loss function is used as follows:

[0137]

[0138] Among them, img is the input image, and ι Describe the input target sequence and input sequence respectively, L is the length of the target sequence, w ι is the weight of the lth token in the sequence.

[0139] Train the model and test it. Based on the built model and the constructed loss function, set the relevant hyperparameters, such as optimizer, learning rate, batch-size, etc., and start training the model. During the test, the nucleus sampling method is used to select the token. When the token output by the model is EOS, it indicates the end of the sequence, and then the generated sequence is converted into a detection box and the corresponding category.

[0140] It can be seen that although the Transformer in this embodiment has good performance in NLP, when it is introduced into CV, when the processing object is an image, due to its own characteristics, the computational complexity and spatial complexity are very large, forcing the model to only use low-resolution image features. The present invention still adopts this approach for the PIX2SEQ model, by introducing a deformable attention mechanism into the transformer, which greatly reduces its complexity, allowing the model to use high-resolution image features and multi-scale feature information, thereby improving the model detection performance.

[0141] The following is an introduction to the target detection model training device provided in an embodiment of the present application. The target detection model training device described below and the target detection model training method described above can refer to each other.

[0142] Please refer to Figure 3, Figure 3 A schematic diagram of the structure of a training device for a target detection model provided in an embodiment of the present application.

[0143] In this embodiment, the device may include:

[0144] The feature extraction module 100 is used to perform feature extraction operations on the image using the backbone network of the model to be trained to obtain a multi-scale feature map;

[0145] A feature encoding module 200, used to encode the multi-scale feature map based on the multi-scale deformable attention encoding module to obtain encoded image features;

[0146] A sequence construction module 300 is used to perform sequence construction processing based on the correct annotation of the image to obtain a target sequence;

[0147] A feature decoding module 400 is used to decode the encoded image features and the target sequence based on the multi-scale deformable attention decoding module to obtain a predicted sequence;

[0148] The parameter updating module 500 is used to update the parameters of the model to be trained based on a preset loss function, a prediction sequence and a correctly labeled target sequence of the image.

[0149] The present application also provides a server, including:

[0150] Memory for storing computer programs;

[0151] A processor is used to implement the steps of the target detection model training method as described in the above embodiment when executing the computer program.

[0152] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the target detection model training method described in the above embodiment are implemented.

[0153] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0154] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0156] The above is a detailed introduction to the training method, training device, server and computer-readable storage medium of a target detection model provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for training a target detection model, characterized in that: include: Use the backbone network of the model to be trained to perform feature extraction on the image to obtain a multi-scale feature map; Encoding the multi-scale feature map based on a multi-scale deformable attention encoding module to obtain encoded image features; Performing sequence construction processing based on the correct annotation of the image to obtain a target sequence; Decoding the encoded image features and the enhanced target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence; wherein the enhanced target sequence is a target sequence after data enhancement and mask processing are performed on the target sequence; Performing parameter updates on the model to be trained based on a preset loss function, the prediction sequence, and a correctly labeled target sequence of the image; The process of constructing a multi-scale deformable attention encoding module includes: Constructing a single-scale deformable attention module, and constructing a multi-scale deformable attention module based on the single-scale deformable attention module, replacing the attention module in the transformer with the multi-scale deformable attention module to obtain a multi-scale deformable attention encoding module; The process of building a multi-scale deformable attention decoding module includes: Replace the cross-attention layer module in the decoder with a multi-scale deformable attention module and keep the self-attention module unchanged to obtain a multi-scale deformable attention decoding module; Performing sequence construction processing based on the correct annotation of the image to obtain a target sequence includes: Performing size normalization processing on the correct marks in the image to obtain a normalized image; performing sequence construction processing on any target in the normalized image to obtain the target sequence; Among them, the formula for constructing a multi-scale deformable attention module based on the single-scale deformable attention module is as follows: ； in, , , represents the input multi-scale feature map, , h is the attention head index, l is the current feature layer, L is the total feature layer, k is the index of the selected keys, Indicates the number of all selected keys required by the current layer. The coefficient of K can ensure that low-level features select more keys, while high-level features select a small number of keys. represents the offset of the kth sample of the lth layer feature in the hth head, represents the weight corresponding to the kth sample of the lth layer feature in the hth head, Represents the coordinates after renormalization of the l-th layer features, Indicates feature points.

2. The training method according to claim 1, characterized in that: Use the backbone network of the model to be trained to perform feature extraction on the image to obtain a multi-scale feature map, including: Performing image enhancement processing on the image to obtain a transformed image; An image feature extraction operation is performed on the transformed image based on the backbone network to obtain the multi-scale feature map.

3. The training method according to claim 1, characterized in that: The multi-scale feature map is encoded based on a multi-scale deformable attention encoding module to obtain encoded image features, including: Acquire image features of each scale from the multi-scale feature map; Encoding the position of each feature point in the image feature to obtain a position code; Fusion of the fusion features of the image features at each scale with the position code to obtain a code input; The encoding input is encoded based on the multi-scale deformable attention encoding module to obtain the encoded image features.

4. The training method according to claim 1, characterized in that: The parameters of the model to be trained are updated based on a preset loss function, the prediction sequence, and the correctly labeled target sequence of the image, including: Constructing a loss function based on the model structure of the model to be trained; Parameters of the model to be trained are updated based on the loss function, the prediction sequence and the correctly labeled target sequence of the image.

5. The training method according to claim 1, characterized in that: Also includes: When the training of the model to be trained is completed, the model to be trained is used as the target detection model.

6. The training method according to claim 1, characterized in that: Also includes: Testing the target detection model based on the labeled data; When the test passes, a test success message is sent.

7. A training device for a target detection model, characterized in that: include: A feature extraction module is used to perform feature extraction operations on the image using the backbone network of the model to be trained to obtain a multi-scale feature map; A feature encoding module, used for encoding the multi-scale feature map based on the multi-scale deformable attention encoding module to obtain encoded image features; A sequence construction module, used for performing sequence construction processing based on the correct annotation of the image to obtain a target sequence; A feature decoding module, used for decoding the encoded image features and the target sequence based on a multi-scale deformable attention decoding module to obtain a predicted sequence; A parameter updating module, used for updating the parameters of the model to be trained based on a preset loss function, the prediction sequence and the correctly labeled target sequence of the image; Among them, the training device of the target detection model is specifically used for: Constructing a single-scale deformable attention module, and constructing a multi-scale deformable attention module based on the single-scale deformable attention module, replacing the attention module in the transformer with the multi-scale deformable attention module to obtain a multi-scale deformable attention encoding module; The training device of the target detection model is specifically used for: Replace the cross-attention layer module in the decoder with a multi-scale deformable attention module and keep the self-attention module unchanged to obtain a multi-scale deformable attention decoding module; Sequence building blocks, specifically for: Performing size normalization processing on the correct marks in the image to obtain a normalized image; performing sequence construction processing on any target in the normalized image to obtain the target sequence; Among them, the formula for constructing a multi-scale deformable attention module based on the single-scale deformable attention module is as follows: ； in, , , represents the input multi-scale feature map, , h is the attention head index, l is the current feature layer, L is the total feature layer, k is the index of the selected keys, Indicates the number of all selected keys required by the current layer. The coefficient of K can ensure that low-level features select more keys, while high-level features select a small number of keys. represents the offset of the kth sample of the lth layer feature in the hth head, represents the weight corresponding to the kth sample of the lth layer feature in the hth head, Represents the coordinates after renormalization of the l-th layer features, Indicates feature points.

8. A server, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for training a target detection model as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the target detection model training method according to any one of claims 1 to 6 are implemented.

Images