Target detection method and device, training method and device, electronic equipment and storage medium

Abstract

A target detection method, a training method, a device, an electronic device, and a storage medium, the target detection method comprising: performing feature extraction on an image to obtain output features of n stages; splicing the output features; encoding the spliced features using an encoder of a Transform network; predicting the encoding result using a prediction head of the Transform network to obtain a candidate box and class prediction parameters, and determining a predicted candidate box and class; optimizing the predicted candidate box and class, comprising: a first step of decoding the candidate box and class using a decoder of the Transform network; a second step of predicting the decoding result using the prediction head to obtain a candidate box and class prediction parameters, and determining an updated candidate box according to the current candidate box prediction parameters and the predicted candidate box, and determining an updated class according to the current class prediction parameters; and repeatedly performing the first step and the second step to obtain a final candidate box and class.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a target detection method, training method, device, electronic device and storage medium. Background Technology

[0002] Object detection is the task of finding all objects of interest in an image and determining their category and location. It is one of the core problems in the field of computer vision. Due to the different appearances, shapes, and poses of various objects, coupled with interference from factors such as lighting and occlusion during imaging, object detection has always been one of the most challenging problems in computer vision.

[0003] Improving the performance of target detection methods is an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a target detection method, training method, apparatus, electronic device, and storage medium to address the problem of how to improve the performance of target detection methods.

[0005] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide a target detection method, comprising:

[0007] A feature extraction network is used to extract features from the image to be detected, resulting in n stages of output features, where n is an integer greater than 1;

[0008] The output features of the n stages are concatenated to obtain the concatenated features;

[0009] The stitched features are encoded using an encoder from a Transform network to obtain the encoding result;

[0010] The prediction head of the Transform network is used to predict the encoding result to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and the predicted candidate box and category are determined based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0011] The predicted candidate boxes and categories are optimized; wherein, the optimization of the predicted candidate boxes and categories includes:

[0012] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0013] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0014] Repeat the first and second steps a preset number of times to obtain the final predicted candidate boxes and categories.

[0015] Optionally, the feature extraction network includes a backbone network and an FPN, and the feature extraction network is used to extract features from the image to be detected, including:

[0016] The image to be detected is input into the backbone network for feature extraction, and the first output features of the n stages of the backbone network are obtained.

[0017] The first output features of the n stages are input into the FPN for feature extraction to obtain the second output features of the n stages of the FPN. The second output features of the n stages are used as the output features of the n stages.

[0018] Optionally, the output features of each stage are represented as [N, C]. f H i W i ], where i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, and C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage;

[0019] The concatenated features are obtained by concatenating the output features of the n stages, including:

[0020] For the output features [N, C] of each stage f H i W i Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ];

[0021] The transposed output features [N, H] i W i C f The transformation is performed to obtain the transformed output features [N, H]. i W i C f ];

[0022] The transformed output features [N, H] of the n stages i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0023] Optionally, the predicted candidate box and category are determined based on the first prediction parameter of the candidate box and the first prediction parameter of the category. Before this, the method further includes: generating position anchors and step size information for the output features of each stage; concatenating the position anchors of the output features of the n stages to obtain concatenated position anchors; and concatenating the step size information of the output features of the n stages to obtain concatenated step size information.

[0024] Determining the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category includes: determining the predicted candidate box based on the first prediction parameter of the candidate box, the stitched position anchor point, and the stitched step size information.

[0025] Optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0026] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0027] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0028] Optionally, determining the predicted candidate boxes and categories based on the first prediction parameters of the candidate boxes and the first prediction parameters of the categories includes:

[0029] The predicted candidate boxes and categories are processed using a nonmaximum suppression method to obtain the M predicted candidate boxes and categories with the highest scores, which are then used as the predicted candidate boxes and categories that need to be optimized.

[0030] Optionally, the updated predicted candidate box is represented as: b1 = r1 + b0;

[0031] Where b1 is the updated predicted candidate box, b0 is the predicted candidate box from the previous side, and r1 is the second prediction parameter of the candidate box.

[0032] Secondly, embodiments of the present invention provide a training method for an object detection network, the object detection network including a feature extraction network and a Transform network, the training method including:

[0033] The feature extraction network is used to extract features from the training image to obtain output features at n stages, where n is an integer greater than 1;

[0034] The output features of the n stages are concatenated to obtain the concatenated features;

[0035] The encoder of the Transform network is used to encode the spliced ​​features to obtain the encoding result;

[0036] The prediction head of the Transform network is used to predict the encoding result to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and the predicted candidate box and category are determined based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0037] Based on the labels of the training images and the predicted candidate boxes and categories, determine the localization loss and classification loss;

[0038] The predicted candidate boxes and categories are optimized; wherein, the optimization of the predicted candidate boxes and categories includes:

[0039] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0040] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0041] The third step: Determine the localization loss and classification loss based on the labels of the training images and the updated predicted candidate boxes and categories;

[0042] Repeat the first, second, and third steps a preset number of times;

[0043] The target detection network is optimized by using all localization and classification losses as the final loss.

[0044] Optionally, the feature extraction network includes a backbone network and an FPN, and using the feature extraction network to extract features from the training image includes:

[0045] The training image is input into the backbone network for feature extraction to obtain the first output features of the n stages of the backbone network;

[0046] The first output features of the n stages are input into the FPN for feature extraction to obtain the second output features of the n stages of the FPN. The second output features of the n stages are used as the output features of the n stages.

[0047] Optionally, the output features of each stage are represented as [N, C]. f H i W i ], where i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, and C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage;

[0048] The concatenated features are obtained by concatenating the output features of the n stages, including:

[0049] For the output features [N, C] of each stage f H i W i Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ];

[0050] The transposed output features [N, H] i W i C f The transformation is performed to obtain the transformed output features [N, H]. i W i C f ];

[0051] The transformed output features [N, H] of the n stages i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0052] Optionally, the predicted candidate box and category are determined based on the first prediction parameter of the candidate box and the first prediction parameter of the category. Before this, the method further includes: generating position anchors and step size information for the output features of each stage; concatenating the position anchors of the output features of the n stages to obtain concatenated position anchors; and concatenating the step size information of the output features of the n stages to obtain concatenated step size information.

[0053] Determining the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category includes: determining the predicted candidate box based on the first prediction parameter of the candidate box, the stitched position anchor point, and the stitched step size information.

[0054] Optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0055] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0056] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0057] Optionally, determining the predicted candidate boxes and categories based on the first prediction parameters of the candidate boxes and the first prediction parameters of the categories includes:

[0058] The predicted candidate boxes and categories are processed using a nonmaximum suppression method to obtain the M predicted candidate boxes and categories with the highest scores, which are then used as the predicted candidate boxes and categories that need to be optimized.

[0059] Optionally, the updated predicted candidate box is represented as: b1 = r1 + b0;

[0060] Where b1 is the updated predicted candidate box, b0 is the predicted candidate box from the previous side, r1 is the second prediction parameter of the candidate box, and r1 is represented as [N, 4, M], where N is the number of images processed simultaneously in a batch, and M is the number of predicted candidate boxes.

[0061] Thirdly, embodiments of the present invention provide a target detection device, comprising:

[0062] The feature extraction module is used to extract features from the image to be detected using a feature extraction network, and obtain output features at n stages, where n is an integer greater than 1;

[0063] The first splicing module is used to splice the output features of the n stages to obtain spliced ​​features;

[0064] The encoding module is used to encode the spliced ​​features using the encoder of the Transform network to obtain the encoding result;

[0065] The prediction module is used to predict the encoding result using the prediction head of the Transform network to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and to determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0066] An optimization module is used to optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0067] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0068] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0069] Repeat the first and second steps a preset number of times to obtain the final predicted candidate boxes and categories.

[0070] Fourthly, embodiments of the present invention provide a training apparatus for an object detection network, the object detection network including a feature extraction network and a Transform network, the training apparatus comprising:

[0071] The feature extraction module is used to extract features from the training image using the feature extraction network to obtain output features at n stages, where n is an integer greater than 1;

[0072] The first splicing module is used to splice the output features of the n stages to obtain spliced ​​features;

[0073] The encoding module is used to encode the spliced ​​features using the encoder of the Transform network to obtain the encoding result;

[0074] The prediction module is used to predict the encoding result using the prediction head of the Transform network to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and to determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0075] The loss determination module is used to determine the localization loss and classification loss based on the labels of the training images and the predicted candidate boxes and categories.

[0076] The first optimization module is used to optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0077] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0078] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0079] The third step: Determine the localization loss and classification loss based on the labels of the training images and the updated predicted candidate boxes and categories;

[0080] Repeat the first, second, and third steps a preset number of times;

[0081] The second optimization module is used to optimize the target detection network by taking all localization and classification losses as the final loss.

[0082] Fifthly, embodiments of the present invention provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the target detection method as described in the first aspect above, or, when the program is executed by the processor, it implements the steps of the target detection network training method as described in the second aspect above.

[0083] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein when executed by a processor, the computer program implements the steps of the target detection method as described in the first aspect above; or, when executed by a processor, the computer program implements the steps of the target detection network training method as described in the second aspect above.

[0084] In this embodiment of the invention, the target detection network uses a structure of feature extraction network + Transform network, wherein the Transform network adopts the basic structure of encoder + decoder. After the encoder outputs the basic target, the decoder fine-tunes it step by step to generate the final predicted candidate box and category, which effectively improves the performance of the target detection network. Attached Figure Description

[0085] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0086] Figure 1 This is a schematic flowchart of the target detection method according to an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of the target detection network according to an embodiment of the present invention;

[0088] Figure 3 This is a flowchart illustrating the training method of the target detection network according to an embodiment of the present invention;

[0089] Figure 4 This is a schematic diagram of the target detection device according to an embodiment of the present invention;

[0090] Figure 5 This is a schematic diagram of the structure of a training device for an object detection network according to an embodiment of the present invention;

[0091] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] Please refer to Figure 1 This invention provides a target detection method for detecting targets using a target detection network. Please refer to [link / reference]. Figure 2 The target detection network has a structure of feature extraction network + Transform network, and the target detection method includes:

[0094] Step 11: Use a feature extraction network to extract features from the image to be detected, and obtain the output features of n stages, where n is an integer greater than 1;

[0095] Step 12: Concatenate the output features of the n stages to obtain concatenated features;

[0096] Step 13: Encode the spliced ​​features using the encoder of the Transform network to obtain the encoding result;

[0097] Step 14: Use the prediction head of the Transform network to predict the encoding result, obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category;

[0098] Step 15: Optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0099] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0100] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0101] Repeat the first and second steps a preset number of times to obtain the final predicted candidate boxes and categories.

[0102] In this embodiment of the invention, the preset number of times can be set according to specific circumstances, such as 6 times. The essence of repeatedly executing the first and second steps is to fine-tune and optimize the candidate boxes and categories predicted in the previous iteration.

[0103] In this embodiment of the invention, the target detection network uses a structure of feature extraction network + Transform network, wherein the Transform network adopts the basic structure of encoder + decoder. After the encoder outputs the basic target, the decoder fine-tunes it step by step to generate the final predicted candidate box and category, which effectively improves the performance of the target detection network.

[0104] In this embodiment of the invention, the type of feature extraction network is not limited. For example, it can be a backbone network, such as SwinTransformV2 or ResNet. The feature extraction network can also be a backbone network and FPN (feature pyramid network). FPN mainly solves the multi-scale problem in object detection. By simply changing the network connections, it significantly improves the performance of small object detection without increasing the computational cost of the original network model.

[0105] In some embodiments of the present invention, optionally, the feature extraction network includes: a backbone network and an FPN. In step 11 above, using the feature extraction network to extract features from the image to be detected includes:

[0106] Step 111: Input the image to be detected into the backbone network for feature extraction to obtain the first output features of the n stages of the backbone network;

[0107] In some embodiments of the present invention, n can be 4. In this step, the first output features of stages 1 to 4 of the backbone network can be obtained.

[0108] Step 112: Input the first output features of the n stages into the FPN for feature extraction to obtain the second output features of the n stages of the FPN, and use the second output features of the n stages as the output features of the n stages.

[0109] In this embodiment of the invention, optionally, the output features of each stage are represented as [N, C] f H i W i ], where i is the sequence number of the stage, assuming n=4, then the value of i is i∈[1,2,3,4], and N is the number of images processed simultaneously in a batch. In this embodiment, if the detection is performed on a single image to be detected, then N is 1, C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage; in this embodiment of the invention, the number of channels C in each stage. f The same, for example, it can be 256.

[0110] In step 12 above, the output features of the n stages are concatenated to obtain concatenated features, including:

[0111] Step 121: Output features [N, C] for each stage f H i W i Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ];

[0112] Step 122: For the transposed output features [N, H] i W i C f The transformation is performed to obtain the transformed output features [N, H]. iW i C f ];

[0113] Step 123: Transform the output features [N, H] of the n stages into... i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0114] In this embodiment of the invention, the purpose of transposing and transforming the output features is to ensure that, during splicing, the features are typically arranged according to [N, H]. i W i C f The images are joined together in the form of ] .

[0115] In this embodiment of the invention, assuming n=4, that is, the output features of the four stages are concatenated, the concatenated feature F can be expressed by the following formula: F=cat([F1,F2,F3,F4],dim=1), where cat is a function used to construct a multidimensional array, dim=1 indicates concatenation in the first dimension, and F1,F2,F3,F4 represent the output features of the four stages respectively.

[0116] In this embodiment of the invention, optionally, in step 13 above, the encoder can be a multi-layer Transform-type structure, such as DeformableTransform, with a total of 6 layers. When processing the concatenated features, the encoder uses feature F (concatenated features) and learnable positional encoding on Q for Q, K, V (V is a vector representing the input features, and Q and K are feature vectors for calculating attention weights, both of which are obtained from the input features).

[0117] In this embodiment of the invention, optionally, the encoding result obtained after the encoder encodes the splicing features can be represented as: [N, H i W i C f ], because the prediction head is usually based on [N, C f H i W i Therefore, before inputting the prediction head, it is necessary to process [N, H] in the form of ], so [N, H] also needs to be processed in the form of ], i W i C f Transpose [N, C] f H i W i ].

[0118] In this embodiment of the invention, optionally, two fully connected networks are used as prediction heads to predict the encoding results and generate candidate bounding box (bbox) prediction parameters and category prediction parameters. The candidate bounding box prediction parameters are not the final candidate bounding boxes; rather, the final candidate bounding boxes need to be determined based on the location anchor points and stride information.

[0119] In this embodiment of the invention, optionally, the candidate box prediction parameter r and the category prediction parameter c can be represented as: [N, 4, H] i W i ] and [N, L, H i W i ], where L is the number of target categories to be predicted.

[0120] In this embodiment of the invention, optionally, step 14 above, before determining the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category, further includes:

[0121] 1) Generate location anchor points and step size information for the output features of each stage;

[0122] In this embodiment of the invention, optionally, the location anchor point can be represented by the following formula: A i =meshgrad(H i W i ); where meshgrid is the function used to generate grid sampling points. The location anchor point of an image at each stage can be represented as [2, H i W i The anchor points of the N images in each stage can be represented as [N, 2, H]. i W i ], where 2 indicates that the number of channels of the anchor point is 2, such as (x, y).

[0123] In this embodiment of the invention, optionally, the step size information can be represented by the following formula: S i =2 i The step size information of an image at each stage can be represented as [1, H]. i W i The step size information of the N images in each stage can be represented as [N, 1, H]. i W i ], where 1 indicates that the number of channels for step size information is a numerical value.

[0124] 2) The position anchor points of the output features of the n stages are spliced ​​together to obtain the spliced ​​position anchor points;

[0125] In this embodiment of the invention, assuming n=4, that is, the position anchor points of the output features of the four stages are concatenated, the concatenated position anchor point A can be represented as: A=cat([A1, A2, A3, A4], dim=2), where cat is a function used to construct a multidimensional array, dim=2 indicates concatenation in the second dimension, and A1, A2, A3, A4 represent the position anchor points of the output features of the four stages respectively.

[0126] 3) The step size information of the output features of the n stages is concatenated to obtain the concatenated step size information;

[0127] In this embodiment of the invention, assuming n=4, that is, the step size information of the output features of the four stages is concatenated, the concatenated step size information S can be expressed as: S=cat([S1, S2, S3, S4], dim=2), where cat is a function used to construct a multidimensional array, dim=2 indicates concatenation in the second dimension, and S1, S2, S3, S4 represent the step size information of the output features of the four stages respectively.

[0128] The process of determining the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category includes: determining the predicted candidate box based on the first prediction parameter of the candidate box, the stitched position anchor point, and the stitched step size information.

[0129] In this embodiment of the invention, optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0130] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0131] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0132] In this embodiment of the invention, optionally, in step 14 above, determining the predicted candidate boxes and categories based on the first prediction parameters of the candidate boxes and the first prediction parameters of the categories includes: processing the predicted candidate boxes and categories using the non-maximum suppression (NMS) method to obtain the M predicted candidate boxes and categories with the highest scores, which are used as the predicted candidate boxes and categories that need to be optimized. The value of M can be set, for example, to 200. The M predicted candidate boxes and categories with the highest scores can be represented as: [N, 4, M].

[0133] In this embodiment of the invention, optionally, before the step of decoding the predicted candidate boxes and categories using the decoder of the Transform network to obtain the decoding result, the method further includes: mapping the predicted candidate boxes and categories to [N,C] using a fully connected network. f [,M], and [N,C] f The transpose of [M, N, M, C] is [N, M, C]. f ,], which serve as the input to the decoder.

[0134] In this embodiment of the invention, optionally, the decoder can be a multi-layer Transform structure, such as DeformableTransform with a total of 6 layers. When the decoder processes the predicted candidate boxes and categories of the input, Q is the sum of B1 and the decoder's learnable query, K and V are F, V is a vector representing the input features, Q and K are feature vectors for calculating attention weights, and they are all obtained from the input features. B1 is the output of the decoder.

[0135] In the second step above, the prediction head of the Transform network is used to predict the decoding result to obtain the candidate box second prediction parameter r1 and the category second prediction parameter c1, wherein the candidate box second prediction parameter r1 and the category second prediction parameter c1 can be represented as [N,4,M] and [N,L,M], respectively.

[0136] In this embodiment of the invention, optionally, the updated predicted candidate box is represented as: b1 = r1 + b0; where b1 is the updated predicted candidate box, b0 is the candidate box predicted on the previous side, and r1 is the second prediction parameter of the candidate box.

[0137] Please refer to Figure 3 This invention also provides a method for training an object detection network, the object detection network including a feature extraction network and a Transform network, the training method including:

[0138] Step 31: Use the feature extraction network to extract features from the training image to obtain output features at n stages, where n is an integer greater than 1;

[0139] Step 32: Concatenate the output features of the n stages to obtain concatenated features;

[0140] Step 33: Encode the spliced ​​features using the encoder of the Transform network to obtain the encoding result;

[0141] Step 34: Use the prediction head of the Transform network to predict the encoding result, obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category;

[0142] Step 35: Determine the localization loss and classification loss based on the labels of the training images and the predicted candidate boxes and categories;

[0143] Step 36: Optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0144] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0145] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0146] The third step: Determine the localization loss and classification loss based on the labels of the training images and the updated predicted candidate boxes and categories;

[0147] Repeat the first, second, and third steps a preset number of times;

[0148] Step 37: Optimize the target detection network by using all localization and classification losses as the final loss.

[0149] In this embodiment of the invention, the preset number of times can be set according to specific circumstances, such as 6 times. The essence of repeatedly executing the first, second, and third steps is to fine-tune and optimize the candidate boxes and categories predicted in the previous step.

[0150] In this embodiment of the invention, the target detection network uses a structure of feature extraction network + Transform network, wherein the Transform network adopts the basic structure of encoder + decoder. After the encoder outputs the basic target, the decoder fine-tunes it step by step to generate the final predicted candidate box and category, which effectively improves the performance of the target detection network.

[0151] In this embodiment of the invention, the type of feature extraction network is not limited. For example, it can be a backbone network, such as SwinTransformV2 or ResNet. The feature extraction network can also be a backbone network and FPN (feature pyramid network). FPN mainly solves the multi-scale problem in object detection. By simply changing the network connections, it significantly improves the performance of small object detection without increasing the computational cost of the original network model.

[0152] In some embodiments of the present invention, optionally, the feature extraction network includes: a backbone network and an FPN, and in step 31 above, using the feature extraction network to extract features from the training image includes:

[0153] Step 311: Input the training image into the backbone network for feature extraction to obtain the first output features of the n stages of the backbone network;

[0154] In some embodiments of the present invention, n can be 4. In this step, the first output features of stages 1 to 4 of the backbone network can be obtained.

[0155] Step 312: Input the first output features of the n stages into the FPN for feature extraction to obtain the second output features of the n stages of the FPN, and use the second output features of the n stages as the output features of the n stages.

[0156] In this embodiment of the invention, optionally, the output features of each stage are represented as [N, C] f H i W i ], where i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, and C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage; In this embodiment of the invention, the number of channels C in each stage... f The same, for example, it can be 256.

[0157] In step 32 above, the output features of the n stages are concatenated to obtain concatenated features, including:

[0158] Step 321: For the output features [N, C] of each stage f H i Wi Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ];

[0159] Step 322: For the transposed output features [N, H] i W i C f The transformation is performed to obtain the transformed output features [N, H]. i W i C f ];

[0160] Step 323: Transform the output features [N, H] of the n stages into... i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0161] In this embodiment of the invention, the purpose of transposing and transforming the output features is to ensure that, during splicing, the features are typically arranged according to [N, H]. i W i C f The images are joined together in the form of ] .

[0162] In this embodiment of the invention, assuming n=4, that is, the output features of the four stages are concatenated, the concatenated feature F can be expressed by the following formula: F=cat([F1,F2,F3,F4],dim=1), where cat is a function used to construct a multidimensional array, dim=1 indicates concatenation in the first dimension, and F1,F2,F3,F4 represent the output features of the four stages respectively.

[0163] In this embodiment of the invention, optionally, in step 13 above, the encoder can be a multi-layer Transform-type structure, such as DeformableTransform, with a total of 6 layers. When processing the concatenated features, the encoder uses feature F (concatenated features) and learnable positional encoding on Q for Q, K, V (V is a vector representing the input features, and Q and K are feature vectors for calculating attention weights, both of which are obtained from the input features).

[0164] In this embodiment of the invention, optionally, the encoding result obtained after the encoder encodes the splicing features can be represented as: [N, H i W i C f ], because the prediction head is usually based on [N, C f H i W iTherefore, before inputting the prediction head, it is necessary to process [N, H] in the form of ], so [N, H] also needs to be processed in the form of ], i W i C f Transpose [N, C] f H i W i ].

[0165] In this embodiment of the invention, optionally, two fully connected networks are used as prediction heads to predict the encoding results and generate candidate bounding box (bbox) prediction parameters and category prediction parameters. The candidate bounding box prediction parameters are not the final candidate bounding boxes; rather, the final candidate bounding boxes need to be determined based on the location anchor points and stride information.

[0166] In this embodiment of the invention, optionally, the candidate box prediction parameter r and the category prediction parameter c can be represented as: [N, 4, H] i W i ] and [N, L, H i W i ], where L is the number of target categories to be predicted.

[0167] In this embodiment of the invention, optionally, step 34 above, which determines the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category, further includes:

[0168] 1) Generate location anchor points and step size information for the output features of each stage;

[0169] In this embodiment of the invention, optionally, the location anchor point can be represented by the following formula: A i =meshgrad(H i W i ); where meshgrid is the function used to generate grid sampling points. The location anchor point of an image at each stage can be represented as [2, H i W i The anchor points of the N images in each stage can be represented as [N, 2, H]. i W i ], where 2 indicates that the number of channels of the anchor point is 2, such as (x, y).

[0170] In this embodiment of the invention, optionally, the step size information can be represented by the following formula: S i =2 i The step size information of an image at each stage can be represented as [1, H]. i W i The step size information of the N images in each stage can be represented as [N, 1, H]. i W i], where 1 indicates that the number of channels for step size information is a numerical value.

[0171] 2) The position anchor points of the output features of the n stages are spliced ​​together to obtain the spliced ​​position anchor points;

[0172] In this embodiment of the invention, assuming n=4, that is, the position anchor points of the output features of the four stages are concatenated, the concatenated position anchor point A can be represented as: A=cat([A1, A2, A3, A4], dim=2), where cat is a function used to construct a multidimensional array, dim=2 indicates concatenation in the second dimension, and A1, A2, A3, A4 represent the position anchor points of the output features of the four stages respectively.

[0173] 3) The step size information of the output features of the n stages is concatenated to obtain the concatenated step size information;

[0174] In this embodiment of the invention, assuming n=4, that is, the step size information of the output features of the four stages is concatenated, the concatenated step size information S can be expressed as: S=cat([S1, S2, S3, S4], dim=2), where cat is a function used to construct a multidimensional array, dim=2 indicates concatenation in the second dimension, and S1, S2, S3, S4 represent the step size information of the output features of the four stages respectively.

[0175] In step 34 above, determining the predicted candidate box and category based on the first prediction parameter of the candidate box and the first prediction parameter of the category includes: determining the predicted candidate box based on the first prediction parameter of the candidate box, the stitched position anchor point, and the stitched step size information.

[0176] In this embodiment of the invention, optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0177] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0178] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0179] In this embodiment of the invention, optionally, in step 34 above, determining the predicted candidate boxes and categories based on the first prediction parameters of the candidate boxes and the first prediction parameters of the categories includes: processing the predicted candidate boxes and categories using a non-maximum suppression method to obtain the M predicted candidate boxes and categories with the highest scores, which are then used as the predicted candidate boxes and categories that need to be optimized. The value of M can be set, for example, to 200. The M predicted candidate boxes and categories with the highest scores can be represented as: [N, 4, M].

[0180] In this embodiment of the invention, in step 35 above, the localization loss and classification loss are determined based on the labels of the training images and the predicted candidate boxes and categories. Optionally, SimOTA can be used to assign labels to the training images, and GIOU loss and cross-entropy loss can be used to determine the localization loss and classification loss respectively.

[0181] In this embodiment of the invention, in step 35, the total loss L0 can be expressed by the following formula:

[0182]

[0183] in, Represents classification loss, λ0 represents the localization loss, and is used to balance the weights of the localization loss and the classification loss.

[0184] In this embodiment of the invention, in the third step described above, the localization loss and classification loss are determined based on the labels of the training images and the updated predicted candidate boxes and categories. Similarly, SimOTA can be used to assign labels to the training images, and GIOU loss and cross-entropy loss can be used to determine the localization loss and classification loss respectively.

[0185] In this embodiment of the invention, the total loss L1 in the third step can be expressed by the following formula:

[0186]

[0187] in, Represents classification loss, λ represents the localization loss, and λ1 is used to balance the weights of the localization loss and the classification loss.

[0188] In this embodiment of the invention, optionally, before the step of decoding the predicted candidate boxes and categories using the decoder of the Transform network to obtain the decoding result, the method further includes: mapping the predicted candidate boxes and categories to [N,C] using a fully connected network. f [,M], and [N,C] f The transpose of [M, N, M, C] is [N, M, C]. f ,], which serve as the input to the decoder.

[0189] In this embodiment of the invention, optionally, the decoder can be a multi-layer Transform structure, such as DeformableTransform with a total of 6 layers. When the decoder processes the predicted candidate boxes and categories of the input, Q is the sum of B1 and the decoder's learnable query, K and V are F, V is a vector representing the input features, Q and K are feature vectors for calculating attention weights, and they are all obtained from the input features. B1 is the output of the decoder.

[0190] In the second step above, the prediction head of the Transform network is used to predict the decoding result to obtain the candidate box second prediction parameter r1 and the category second prediction parameter c1, wherein the candidate box second prediction parameter r1 and the category second prediction parameter c1 can be represented as [N,4,M] and [N,L,M], respectively.

[0191] In this embodiment of the invention, optionally, the updated predicted candidate box is represented as: b1 = r1 + b0; where b1 is the updated predicted candidate box, b0 is the candidate box predicted on the previous side, r1 is the second prediction parameter of the candidate box, and r1 is represented as [N, 4, M], where N is the number of images processed simultaneously in a batch, and M is the number of predicted candidate boxes.

[0192] In this embodiment of the invention, optionally, in step 37 above, the optimizer AdamW can be used to optimize the target detection network.

[0193] Please refer to Figure 4 This invention also provides a target detection device 40, comprising:

[0194] The feature extraction module 41 is used to extract features from the image to be detected using a feature extraction network, and obtain output features in n stages, where n is an integer greater than 1;

[0195] The first splicing module 42 is used to splice the output features of the n stages to obtain spliced ​​features;

[0196] Encoding module 43 is used to encode the spliced ​​features using the encoder of the Transform network to obtain the encoding result;

[0197] The prediction module 44 is used to predict the encoding result using the prediction head of the Transform network to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and to determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0198] Optimization module 45 is used to optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0199] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0200] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0201] Repeat the first and second steps a preset number of times to obtain the final predicted candidate boxes and categories.

[0202] Optionally, the feature extraction network includes a backbone network and an FPN. The feature extraction module 41 is used to input the image to be detected into the backbone network for feature extraction to obtain the first output features of the backbone network at n stages; input the first output features of the n stages into the FPN for feature extraction to obtain the second output features of the FPN at n stages; and use the second output features of the n stages as the output features of the n stages.

[0203] Optionally, the output features of each stage are represented as [N, C]. f H i W i ], where i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, and C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage;

[0204] The first splicing module 42 is used to process the output features [N, C] of each stage. f H i W i Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ]; for the transposed output features [N, H i W i C f The transformation is performed to obtain the transformed output features [N, H]. i W i C fThe transformed output features [N, H] of the n stages are... i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0205] Optionally, the target detection device 40 further includes:

[0206] The generation module is used to generate location anchors and step size information for the output features of each stage.

[0207] The second splicing module is used to splice the position anchor points of the output features of the n stages to obtain the spliced ​​position anchor points;

[0208] The third splicing module is used to splice the step size information of the output features of the n stages to obtain the spliced ​​step size information;

[0209] The prediction module 44 is used to determine the predicted candidate box based on the first prediction parameters of the candidate box, the spliced ​​position anchor point, and the spliced ​​step size information.

[0210] Optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0211] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0212] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0213] Optionally, the prediction module 44 is used to process the predicted candidate boxes and categories using a non-maximum suppression method to obtain the M predicted candidate boxes and categories with the highest scores, which are then used as the predicted candidate boxes and categories that need to be optimized.

[0214] Optionally, the updated predicted candidate box is represented as: b1 = r1 + b0;

[0215] Where b1 is the updated predicted candidate box, b0 is the predicted candidate box from the previous side, and r1 is the second prediction parameter of the candidate box.

[0216] Please refer to Figure 5 This invention also provides a training apparatus 50 for an object detection network, the object detection network including a feature extraction network and a Transform network, the training apparatus 50 for the object detection network including:

[0217] The feature extraction module 51 is used to extract features from the training image using the feature extraction network to obtain output features at n stages, where n is an integer greater than 1;

[0218] The first splicing module 52 is used to splice the output features of the n stages to obtain spliced ​​features;

[0219] Encoding module 53 is used to encode the splicing features using the encoder of the Transform network to obtain an encoding result;

[0220] The prediction module 54 is used to predict the encoding result using the prediction head of the Transform network to obtain the first prediction parameters of the candidate box and the first prediction parameters of the category, and to determine the predicted candidate box and category based on the first prediction parameters of the candidate box and the first prediction parameters of the category.

[0221] The loss determination module 55 is used to determine the localization loss and classification loss based on the labels of the training images and the predicted candidate boxes and categories.

[0222] The first optimization module 56 is used to optimize the predicted candidate boxes and categories; wherein, optimizing the predicted candidate boxes and categories includes:

[0223] First step: Use the decoder of the Transform network to decode the predicted candidate boxes and categories to obtain the decoding results;

[0224] The second step is to use the prediction head of the Transform network to predict the decoding result, obtain the second prediction parameters of the candidate box and the second prediction parameters of the category, and determine the updated predicted candidate box based on the second prediction parameters of the candidate box and the previously predicted candidate box, and determine the updated predicted category based on the second prediction parameters of the category.

[0225] The third step: Determine the localization loss and classification loss based on the labels of the training images and the updated predicted candidate boxes and categories;

[0226] Repeat the first, second, and third steps a preset number of times;

[0227] The second optimization module 57 is used to optimize the target detection network by taking all localization losses and classification losses as the final losses.

[0228] Optionally, the feature extraction network includes a backbone network and an FPN. The feature extraction module 51 is used to input the training image into the backbone network for feature extraction to obtain the first output features of the backbone network at n stages; input the first output features of the n stages into the FPN for feature extraction to obtain the second output features of the FPN at n stages; and use the second output features of the n stages as the output features of the n stages.

[0229] Optionally, the output features of each stage are represented as [N, C]. f H i W i ], where i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, and C f H represents the number of channels. i W represents the height of the output feature in the i-th stage. i The width of the output feature in the i-th stage;

[0230] The first splicing module 52 is used to process the output features [N, C] of each stage. f H i W i Transpose the given values ​​to obtain the transposed output features [N, H]. i W i C f ]; for the transposed output features [N, H i W i C f The transformation is performed to obtain the transformed output features [N, H]. i W i C f The transformed output features [N, H] of the n stages are... i W i C f The splicing is performed in the first dimension to obtain the spliced ​​features.

[0231] Optionally, the training device 50 for the object detection network further includes:

[0232] The generation module is used to generate location anchors and step size information for the output features of each stage.

[0233] The second splicing module is used to splice the position anchor points of the output features of the n stages to obtain the spliced ​​position anchor points;

[0234] The third splicing module is used to splice the step size information of the output features of the n stages to obtain the spliced ​​step size information;

[0235] The prediction module 54 is used to determine the predicted candidate box based on the first prediction parameters of the candidate box, the spliced ​​position anchor point, and the spliced ​​step size information.

[0236] Optionally, the position of the predicted candidate box is represented as: b0[:,:2]=(r[:,:2]+A)*S;

[0237] The size of the predicted candidate box is represented as: b0[:,2:]=e r[:,2:] *S;

[0238] Where b0[:,:2] is the predicted candidate box position, b0[:,2:] is the predicted candidate box size, and r is the first prediction parameter of the candidate box, where r is represented as [N, 4, H]. i W i ], i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, 4 is the number of channels, H i W represents the height of the output feature in the i-th stage. i Let A be the width of the output feature in the i-th stage, A be the anchor point after concatenation, and S be the step size information after concatenation.

[0239] Optionally, the prediction module 54 is used to process the predicted candidate boxes and categories using a non-maximum suppression method to obtain the M predicted candidate boxes and categories with the highest scores, which are then used as the predicted candidate boxes and categories that need to be optimized.

[0240] Optionally, the updated predicted candidate box is represented as: b1 = r1 + b0;

[0241] Where b1 is the updated predicted candidate box, b0 is the predicted candidate box from the previous side, r1 is the second prediction parameter of the candidate box, and r1 is represented as [N, 4, M], where N is the number of images processed simultaneously in a batch, and M is the number of predicted candidate boxes.

[0242] Please refer to Figure 6 The present invention also provides an electronic device 60, including a processor 61, a memory 62, and a computer program stored in the memory 62 and executable on the processor 61. When the computer program is executed by the processor 61, it implements the various processes of the above-described target detection method embodiments, or, when the computer program is executed by the processor 61, it implements the various processes of the above-described target detection network training method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0243] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described target detection method embodiments, or, when executed by a processor, implements the various processes of the above-described target detection network training method embodiments, achieving the same technical effect. To avoid repetition, further details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0244] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0245] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0246] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A target detection method characterized by, The method comprises the following steps: using a feature extraction network to extract features of a to-be-detected image to obtain output features of n stages, n being an integer greater than 1; splicing the output features of the n stages to obtain spliced features; using an encoder of a Transformer network to encode the spliced features to obtain an encoding result; using a prediction head of the Transformer network to predict the encoding result to obtain candidate box first prediction parameters and category first prediction parameters, and determining predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters; optimizing the predicted candidate boxes and categories; wherein optimizing the predicted candidate boxes and categories comprises: a first step: using a decoder of the Transformer network to decode the predicted candidate boxes and categories to obtain a decoding result; wherein the decoder is a multi-layer Transformer structure; a second step: using the prediction head of the Transformer network to predict the decoding result to obtain candidate box second prediction parameters and category second prediction parameters, and determining updated predicted candidate boxes according to the candidate box second prediction parameters and the candidate boxes predicted last time, and determining updated predicted categories according to the category second prediction parameters; repeating the first step and the second step for a preset number of times to obtain final predicted candidate boxes and categories; wherein the output feature representation of each stage is denoted as wherein i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, is the number of channels, is the height of the output feature of the i-th stage, is the width of the output feature of the i-th stage; splicing the output features of the n stages to obtain spliced features comprises: output features for each stage transpose to obtain transposed output features ; on the transposed output features transformed output features ; concatenating the transformed output features of the n stages concatenating in the first dimension to obtain a concatenated feature 2. The method of claim 1, wherein, the feature extraction network comprises a backbone network and an FPN, and using the feature extraction network to extract features of a to-be-detected image comprises: inputting the to-be-detected image into the backbone network for feature extraction to obtain first output features of n stages of the backbone network; inputting the first output features of the n stages into the FPN for feature extraction to obtain second output features of n stages of the FPN, and taking the second output features of the n stages as the output features of the n stages.

3. The method of claim 1, wherein: determining predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters further comprises: generating position anchor points and step length information for the output features of each stage; splicing the position anchor points of the output features of the n stages to obtain spliced position anchor points; and splicing the step length information of the output features of the n stages to obtain spliced step length information; determining predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters comprises: determining predicted candidate boxes according to the candidate box first prediction parameters, the spliced position anchor points, and the spliced step length information.

4. The method of claim 3, wherein: The position of the predicted candidate box is represented as: ; The size of the predicted candidate box is represented as: ; in, For the predicted candidate box locations, The size of the predicted candidate box. The first prediction parameter for the candidate box. Represented as Where i is the stage number, N is the number of images processed simultaneously in a batch, and 4 is the number of channels. The height of the output feature in the i-th stage. Let be the width of the output feature of the i-th stage. These are the anchor points for the assembled position. This is the step size information after splicing.

5. The method of claim 1, wherein, determining predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters comprises: The non-maximum suppression method is used to process the predicted candidate frame and the category, to obtain M predicted candidate frames and categories with the highest scores as predicted candidate frames and categories that need to be optimized.

6. The method of claim 3, wherein, The updated predicted candidate box is represented as: ; wherein, is an updated predicted candidate box, is a last side predicted candidate box, is a candidate box second prediction parameter.

7. A method for training a target detection network, the method comprising: The target detection network comprises a feature extraction network and a Transformer network, and the training method comprises: The feature extraction network is used to perform feature extraction on a training image, to obtain output features of n stages, n being an integer greater than 1; The output features of the n stages are spliced to obtain spliced features; An encoder of the Transformer network is used to encode the spliced features, to obtain an encoding result; A prediction head of the Transformer network is used to predict the encoding result, to obtain candidate frame first prediction parameters and category first prediction parameters, and to determine predicted candidate frames and categories according to the candidate frame first prediction parameters and the category first prediction parameters; According to the label of the training image and the predicted candidate frame and the category, a positioning loss and a classification loss are determined; The predicted candidate frame and the category are optimized, wherein the optimization of the predicted candidate frame and the category comprises: A first step: a decoder of the Transformer network is used to decode the predicted candidate frame and the category, to obtain a decoding result; wherein the decoder is a multi-layer Transformer type structure; A second step: a prediction head of the Transformer network is used to predict the decoding result, to obtain candidate frame second prediction parameters and category second prediction parameters, and to determine updated predicted candidate frames according to the candidate frame second prediction parameters and the last predicted candidate frame, and to determine updated predicted categories according to the category second prediction parameters; A third step: according to the label of the training image and the updated predicted candidate frame and the category, a positioning loss and a classification loss are determined; The first step, the second step and the third step are repeatedly executed for a preset number of times; All the positioning losses and the classification losses are taken as final losses, and the target detection network is optimized; wherein the output feature of each stage is represented as wherein i is the serial number of the stage, N is the number of images processed simultaneously in a batch, is the number of channels, is the height of the output feature of the i-th stage, is the width of the output feature of the i-th stage; The output features of the n stages are spliced to obtain spliced features, comprising: output features for each stage transpose to get transposed output features ; transforming the transposed output features to obtain transformed output features ; concatenating the transformed output features of the n stages concatenating in the first dimension to obtain a concatenated feature 8. The method of claim 7, wherein, According to the candidate frame first prediction parameters and the category first prediction parameters, the predicted candidate frame and the category are determined, comprising: The non-maximum suppression method is used to process the predicted candidate frame and the category, to obtain M predicted candidate frames and categories with the highest scores as predicted candidate frames and categories that need to be optimized.

9. The method of claim 8, wherein, The updated predicted candidate box is represented as: ; wherein, is an updated prediction candidate box, is a last side prediction candidate box, is a candidate box second prediction parameter, is represented as N is the number of images processed simultaneously in a batch, and M is the number of prediction candidate boxes.

10. A target detection apparatus characterized by comprising: comprising: a feature extraction module, configured to use a feature extraction network to perform feature extraction on a to-be-detected image, to obtain output features of n stages, n being an integer greater than 1; a first splicing module, configured to splice the output features of the n stages to obtain spliced features; an encoding module, configured to use an encoder of a Transformer network to encode the spliced features, to obtain an encoding result; a prediction module configured to predict the encoding result using a prediction head of the Transformer network to obtain candidate box first prediction parameters and category first prediction parameters, and determine predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters; an optimization module configured to optimize the predicted candidate boxes and categories, wherein optimizing the predicted candidate boxes and categories comprises: a first step of decoding the predicted candidate boxes and categories using a decoder of the Transformer network to obtain decoding results, wherein the decoder is a multi-layer Transformer structure; a second step of predicting the decoding results using the prediction head of the Transformer network to obtain candidate box second prediction parameters and category second prediction parameters, and determining updated predicted candidate boxes according to the candidate box second prediction parameters and the candidate boxes predicted in the last time, and determining updated predicted categories according to the category second prediction parameters; repeating the first step and the second step for a preset number of times to obtain final predicted candidate boxes and categories; wherein the output feature representation of each stage is represented as wherein i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, is the number of channels, is the height of the output feature of the i-th stage, is the width of the output feature of the i-th stage; splicing the output features of the n stages to obtain spliced features, comprising: output features for each stage transpose to obtain transposed output features ; on the transposed output features transformed output features ; concatenating the transformed output features of the n stages concatenating in the first dimension to obtain a concatenated feature 11. A training device for an object detection network, characterized in that, the target detection network comprises a feature extraction network and a Transformer network, and the training device comprises: a feature extraction module configured to extract features of a training image using the feature extraction network to obtain n-stage output features, n being an integer greater than 1; a first splicing module configured to splice the n-stage output features to obtain spliced features; an encoding module configured to encode the spliced features using an encoder of the Transformer network to obtain an encoding result; a prediction module configured to predict the encoding result using a prediction head of the Transformer network to obtain candidate box first prediction parameters and category first prediction parameters, and determine predicted candidate boxes and categories according to the candidate box first prediction parameters and the category first prediction parameters; a loss determination module configured to determine a positioning loss and a classification loss according to labels of the training image and the predicted candidate boxes and categories; a first optimization module configured to optimize the predicted candidate boxes and categories, wherein optimizing the predicted candidate boxes and categories comprises: a first step of decoding the predicted candidate boxes and categories using a decoder of the Transformer network to obtain decoding results, wherein the decoder is a multi-layer Transformer structure; a second step of predicting the decoding results using the prediction head of the Transformer network to obtain candidate box second prediction parameters and category second prediction parameters, and determining updated predicted candidate boxes according to the candidate box second prediction parameters and the candidate boxes predicted in the last time, and determining updated predicted categories according to the category second prediction parameters; a third step of determining a positioning loss and a classification loss according to the labels of the training image and the updated predicted candidate boxes and categories; repeating the first step, the second step and the third step for a preset number of times; a second optimization module configured to optimize the target detection network by taking all the positioning losses and the classification loss as a final loss; wherein the output feature representation of each stage is denoted as wherein i is the sequence number of the stage, N is the number of images processed simultaneously in a batch, is the number of channels, is the height of the output feature of the i-th stage, is the width of the output feature of the i-th stage; concatenating the output features of the n stages to obtain a concatenated feature, including: output features for each stage transpose to obtain transposed output features ; on the transposed output features transformed output features ; concatenating the transformed output features of the n stages concatenating in the first dimension to obtain a concatenated feature 12. An electronic device, comprising: including: a processor, a memory, and a program stored in the memory and executable on the processor, the program being executed by the processor to implement the steps of the target detection method according to any one of claims 1 to 6, or the program being executed by the processor to implement the steps of the training method of the target detection network according to any one of claims 7 to 9.

13. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executable on the processor to implement the steps of the target detection method according to any one of claims 1 to 6; or the computer program is executable on the processor to implement the steps of the training method of the target detection network according to any one of claims 7 to 9.

Images