A target detection method, device and storage medium based on DETR-YOLO

Abstract

The present application provides a DETR-YOLO-based target detection method, device, and storage medium. The DETR-YOLO-based target detection method obtains an image to be detected; inputs the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence; finally, the target in the image to be detected is detected according to the DETR-YOLO model. The DETR-YOLO-based target detection method detects targets using the DETR-YOLO model. Since the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence, the DETR network can output all prediction results at once during feature map processing, thereby improving the model training efficiency and target detection efficiency.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically to a target detection method, device, and storage medium based on DETR-YOLO. Background Art

[0002] With the continuous advancement of side-scan sonar detection technology, the use of side-scan sonar to search for wrecked ships has become an important part of hydrographic obstacle verification and maritime search and rescue.

[0003] After the image is collected through side-scan sonar detection, the image is identified and detected through the target detection algorithm, and the corresponding rectangular frame and classification information are output. The output rectangular frame and classification information are then used for obstacle verification and maritime search and rescue.

[0004] The Faster R-CNN model is an existing model that uses a target detection algorithm to automatically detect submarine shipwrecks using side-scan sonar. However, this model suffers from technical issues such as low training efficiency and low detection efficiency. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a target detection method, device and storage medium based on DETR-YOLO to improve the technical problem of low training and detection efficiency in the above-mentioned target detection model.

[0006] To achieve the above objectives, this application provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a target detection method based on DETR-YOLO, the method comprising:

[0008] Obtain the image to be detected;

[0009] Inputting the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence;

[0010] The target in the image to be detected is detected according to the DETR-YOLO model.

[0011] In the above scheme, the DETR-YOLO-based target detection method obtains an image to be detected; inputs the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence; finally, the target in the image to be detected is detected based on the DETR-YOLO model. The DETR-YOLO-based target detection method detects targets using the DETR-YOLO model. Since the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence, the DETR network can output all prediction results at once during feature map processing, thereby improving the model training efficiency and target detection efficiency.

[0012] Optionally, detecting the target in the image to be detected based on the DETR-YOLO model includes: performing feature extraction on the input image to be detected through the baseline network to obtain a first feature map, and sending the first feature map to the DETR network; performing target detection on the received first feature map through the DETR network to obtain a first target detection result, and sending the first target detection result to the Neck network; performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result; and determining the target in the image to be detected based on the second target detection result.

[0013] Optionally, the DETR-YOLO model may also include an input end and an output end; the input end is connected to the baseline network, and the output end is connected to the Neck network; the input image to be detected is received through the input end, and the image to be detected is sent to the baseline network; the output end receives the second target detection result obtained by the Neck network, and the output end performs a vector convolution operation on the second target detection result to obtain the target in the image to be detected, and outputs the target.

[0014] Optionally, the DETR-YOLO model also includes a SENet network, which is arranged between the baseline network and the DETR network; the feature extraction of the input image to be detected through the baseline network to obtain a first feature map, and sending the first feature map to the DETR network includes: extracting features of the input image to be detected through the baseline network to obtain a second feature map, and sending the second feature map to the SENet network; receiving the second feature map through the SENet network, compressing and excitation processing the second feature map to obtain a first feature map, and sending the first feature map to the DETR network.

[0015] In the above solution, by adding the SENet network to the DETR-YOLO model and optimizing the model structure through the SENet attention mechanism, the model can automatically learn the importance of each feature channel and assign corresponding weights based on global information. This can enhance the learning of beneficial features while suppressing the learning of redundant features, thereby making feature learning more targeted and improving the detection efficiency and accuracy of the DETR-YOLO model.

[0016] Optionally, the DETR-YOLO model includes receptive fields of 4*4, 8*8, 16*16 and 32*32; the performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain the second target detection result includes: receiving the first target detection result through the Neck network; wherein the first target detection result includes a first 4*4 target detection result, a first 8*8 target detection result, a first 16*16 target detection result and a first 32*32 target detection result; performing sampling and fusion operations on the first 4*4 target detection result, the first 8*8 target detection result, the first 16*16 target detection result and the first 32*32 target detection result respectively through the Neck network to obtain the second target detection result.

[0017] In the above scheme, by increasing the 4*4 receptive field, smaller-scale targets can be detected, and sampling and fusion of multiple-scale targets such as 4*4, 8*8, 16*16 and 32*32 can be performed, which improves the model's ability to extract and fuse features of small-scale targets, further improving the detection accuracy of the DETR-YOLO model.

[0018] Optionally, performing target detection on the received first feature map through the DETR network to obtain a first target detection result includes: receiving the first feature map through the DETR network, performing target detection on the first feature map, and obtaining a first target prediction box; screening the first target prediction box through the DETR network using weighted box fusion to generate a first detection box; and determining the first target detection result based on the first detection box through the DETR network.

[0019] In the above scheme, the first target prediction frame is screened through weighted frame fusion to generate the first detection frame, thereby improving the positioning accuracy of the first detection frame, that is, improving the accuracy of the first target detection result, thereby reducing the missed alarm rate and false alarm rate of the overall model.

[0020] Optionally, the DETR network is used to filter the first target prediction frame using weighted frame fusion to generate a first detection frame, including: determining the weight of the first target prediction frame according to the confidence of the first target prediction frame; determining the first fusion confidence of the first fusion frame according to the first target prediction frame weight; if the first fusion confidence is greater than a first preset confidence threshold, determining the position coordinates of the upper left corner and lower right corner of the first fusion frame according to the upper left corner and lower right corner coordinate formulas; wherein the upper left corner coordinate formula is The formula for the coordinates of the lower right corner is Among them, c i is the confidence of the first target prediction box of the i-th object, (x1 i , y1 i ) is the coordinate of the upper left corner of the i-th first target prediction box, (x2 i , y2 i ) is the coordinate of the lower right corner of the i-th first target prediction box, (x1 m , y1 m ) is the upper left corner position coordinate of the mth first fusion frame, (x2 m , y2 m ) is the lower right corner position coordinate of the mth first fused frame; and the first detection frame is generated according to the upper left corner position coordinate and the lower right corner position coordinate of the first fused frame.

[0021] In the above scheme, the confidence and position coordinates of the first target prediction frame are obtained according to the confidence and position coordinates of the first detection frame and the weighted frame fusion method is adopted, which improves the confidence and positioning accuracy of the first detection frame, that is, improves the accuracy of the first target detection result, thereby reducing the missed alarm rate and false alarm rate of the overall model.

[0022] Optionally, performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result includes: receiving the first target detection result through the Neck network, performing feature fusion and feature extraction on the first target detection result to obtain a second target prediction box; screening the second target prediction box through the Neck network using weighted box fusion to generate a second detection box; and determining the second target detection result based on the second detection box through the Neck network.

[0023] In the above scheme, the second target prediction frame is screened through weighted frame fusion to generate a second detection frame, thereby improving the positioning accuracy of the second detection frame, that is, improving the accuracy of the second target detection result, thereby reducing the missed alarm rate and false alarm rate of the overall model.

[0024] Optionally, the training process of the DETR-YOLO model includes: obtaining multiple training images and known targets in the training images; inputting the training images into the model to be trained to obtain the target detection results of the model to be trained; and optimizing the internal parameters of the model to be trained according to the known targets and the target detection results to obtain a trained DETR-YOLO model.

[0025] In a second aspect, an embodiment of the present application provides a target detection device based on DETR-YOLO, the device comprising:

[0026] An image acquisition module, wherein the image acquisition module is used to acquire an image to be detected;

[0027] An image input module, wherein the image input module is used to input the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence;

[0028] An image detection module is used to detect the target in the image to be detected based on the DETR-YOLO model.

[0029] In the above scheme, the DETR-YOLO-based target detection device includes an image acquisition module, an image input module, and an image detection module. The DETR-YOLO-based target detection device acquires an image to be detected; inputs the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence; finally, the target in the image to be detected is detected based on the DETR-YOLO model. The DETR-YOLO-based target detection device detects targets using the DETR-YOLO model. Since the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence, and the DETR network can output all prediction results at once during feature map processing, the model training efficiency and target detection efficiency are improved.

[0030] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are read and executed by a processor, the method provided by any possible implementation of the first aspect is executed.

[0031] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a target detection method based on DETR-YOLO provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the process of target detection using the DETR-YOLO model provided in this application;

[0035] Figure 3 A schematic diagram of another process for target detection using the DETR-YOLO model provided in this application;

[0036] Figure 4 A schematic diagram of a feature fusion process provided by this application;

[0037] Figure 5 A schematic diagram of the structure of a target detection device based on DETR-YOLO provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0039] The terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0040] The terms "first," "second," etc. are only used to distinguish one entity or operation from another entity or operation, and are not to be understood as indicating or implying relative importance, nor are they to be understood as requiring or implying any actual relationship or order between these entities or operations.

[0041] In response to the shortcomings of the existing technology, the embodiment of the present application provides a target detection method based on DETR-YOLO, which is used to improve the technical problems of low training and detection efficiency in the above-mentioned target detection model.

[0042] Please refer to Figure 1 , Figure 1 A flowchart of a target detection method based on DETR-YOLO provided in an embodiment of the present application, the target detection method based on DETR-YOLO includes:

[0043] Step 101: Acquire an image to be detected;

[0044] Step 102: input the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence;

[0045] Step 103: Detect the target in the image to be detected according to the DETR-YOLO model.

[0046] Among them, in step 101, the source of the image to be detected can be image data measured by side-scan sonar equipment or other equipment in areas such as the East China Sea, the Yellow Sea and inland lakes in my country, or other image data that requires target detection.

[0047] In step 102, the YOLO in the DETR-YOLO model can be a YOLOv4 or YOLOv5 model, etc., the baseline network refers to the Backbone part in the existing YOLO model, the DETR (Detection Transformer) network consists of three parts: Encoder, Decoder, and Prediction, and the Neck network refers to the Neck part in the existing YOLO model.

[0048] In step 103, the target in the image to be detected may be a wreck, debris, or a navigation log, etc., which is not specifically limited in this application.

[0049] As can be seen from the above, the present application provides a target detection method based on DETR-YOLO, which obtains an image to be detected; inputs the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence; finally, the target in the image to be detected is detected according to the DETR-YOLO model. The target detection method based on DETR-YOLO detects the target through the DETR-YOLO model. Since the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence, and the DETR network can output all prediction results at once during the feature map processing process, the training efficiency of the model and the target detection efficiency are improved.

[0050] Please refer to Figure 2 , Figure 2 A schematic diagram of the process of target detection using the DETR-YOLO model provided in this application.

[0051] In some optional embodiments, step 103, detecting the target in the image to be detected according to the DETR-YOLO model, includes: step 1031, performing feature extraction on the input image to be detected through the baseline network to obtain a first feature map, and sending the first feature map to the DETR network; step 1032, performing target detection on the received first feature map through the DETR network to obtain a first target detection result, and sending the first target detection result to the Neck network; step 1033, performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result; step 1034, determining the target in the image to be detected according to the second target detection result.

[0052] The step of extracting features of the input image to be detected through the reference network to obtain the first feature map may be: using a conventional convolutional neural network (CNN) to learn features of the input image to be detected to obtain the first feature map.

[0053] Among them, the DETR (Detection Transformer) network consists of three parts: Encoder, Decoder and Prediction. The DETR network performs target detection on the first feature map received to obtain a first target detection result. It can be: the Encoder part of the DETR network performs position encoding on the first feature map, the Decoder part performs feature extraction on the first feature map after position encoding output by the Encoder part, and embeds a small number of fixed positions into Object Queries; then the output of the Decoder part is passed to the Feed Forward Network (FFN) of the Prediction part, and the FFN performs network detection on the output of the Decoder part to obtain the first target detection result, which includes: the category and position of the target or the no-target class.

[0054] The encoder performs positional encoding on the first feature map, including dimensional compression. Specifically, the encoder applies a 1x1 convolution operation to the CxHxW-dimensional feature map, compressing the number of channels C to d, resulting in a dxHxW-dimensional first sub-feature map. Next, the encoder performs sequence conversion on the first sub-feature map, compressing the spatial dimensions HxW to HW, resulting in a 2D dxHW second sub-feature map. Positional encoding is then applied to the second sub-feature map.

[0055] The encoder can contain 6 layers, each containing 8 self-attention modules and FFN; the decoder can also contain 6 layers, each containing 8 self-attention modules, 8 co-attention modules, and FFN. The introduction of the attention module in the DETR network allows the DETR-YOLO model to focus on the valid parts of the image to be detected, improving the targeted learning of the model's target features.

[0056] Please refer to Figure 3 , Figure 3 This is another flowchart of target detection using the DETR-YOLO model provided in this application.

[0057] In some optional embodiments, the DETR-YOLO model also includes a SENet network, which is arranged between the baseline network and the DETR network; step 1031, performing feature extraction on the input image to be detected through the baseline network to obtain a first feature map, and sending the first feature map to the DETR network, including: step 10311, performing feature extraction on the input image to be detected through the baseline network to obtain a second feature map, and sending the second feature map to the SENet network; step 10312, receiving the second feature map through the SENet network, compressing and excitation processing on the second feature map, obtaining a first feature map, and sending the first feature map to the DETR network.

[0058] The SENet network can be divided into two parts: Squeeze and Excitation. The Squeeze part is used to perform global average pooling. The corresponding feature map is compressed in one dimension, that is, the feature map of W*H*C is compressed into a feature map of 1*1*C; where W*H represents the width and height of the feature map, C is the number of channels, and u a (j, k) is the element at position (j, k) of the a-th channel, and a∈C. The Excitation part is used to extract the relationship between each channel through Ex=σ(g(z,W))=σ(W2δ(W1,z)); among them, the Excitation part adopts the gating mechanism in Sigmoid, and reduces the channel to its original value by introducing the fully connected layer FC1 and the parameter W1. After being activated by the ReLU function (δ), the fully connected layer FC2 is used to restore the channel to its original number with the parameter W2, and finally the channel weights are generated by the Sigmoid function (σ); where r is the dimensionality reduction ratio, which can be 16 or other reasonable values. The weight values ​​of each channel are weighted to the corresponding feature channel through the Scale operation to obtain the final output X, X a =F scale (u a )=u a ×Sq a ; Among them, X a is the output of the ath feature channel.

[0059] In some optional embodiments, the DETR-YOLO model includes receptive fields of 4*4, 8*8, 16*16 and 32*32; the feature fusion and feature extraction are performed on the first target detection result received by the Neck network to obtain the second target detection result, including: receiving the first target detection result through the Neck network; wherein the first target detection result includes a first 4*4 target detection result, a first 8*8 target detection result, a first 16*16 target detection result and a first 32*32 target detection result; the first 4*4 target detection result, the first 8*8 target detection result, the first 16*16 target detection result and the first 32*32 target detection result are respectively sampled and fused by the Neck network to obtain the second target detection result.

[0060] Among them, a 4*4 receptive field can be obtained by adding a detection layer. Specifically, a 152*152 feature map can be obtained by downsampling a 3*3 convolution kernel with a step size of 2, and then a 4*4 receptive field can be obtained.

[0061] Among them, adding a detection layer will deepen the complexity of the DETR-YOLO model to a certain extent. The CSP (Cross Stage Paritial) structure can be used to merge the results under the cross-stage structure through the stacking of residual structures and the synchronous operation of convolution to achieve the integration of gradient changes on the feature map, thereby reducing the computational bottleneck and memory cost on the basis of enhancing the learning ability of the DETR-YOLO model.

[0062] Please refer to Figure 4 , Figure 4 A schematic diagram of the feature fusion process provided in this application.

[0063] Among them, the feature fusion and feature extraction of the received first target detection result by the Neck network to obtain the second target detection result specifically includes: the Neck network receives the first target detection result; wherein the first target detection result includes a first 4*4 target detection result, a first 8*8 target detection result, a first 16*16 target detection result and a first 32*32 target detection result; the Neck network performs feature fusion on the first 4*4 target detection result and the first 8*8 target detection result to obtain a first fused 8*8 target detection result; the Neck network performs feature fusion on the first 8*8 target detection result and the first 16*16 target detection result to obtain a first fused 16*16 target detection result; the Neck network performs feature fusion on the first 16*16 target detection result and the first 32*32 target detection result to obtain a first fused 32*32 target detection result; the Neck network downsamples the first fused 32*32 target detection result and performs feature fusion on the first fused 16*16 target detection result to obtain a second fused 16*16 target detection result; the Neck network downsamples the second fused 16*16 target detection result and performs feature fusion on the first fused 8*8 target detection result to obtain a second fused 8*8 target detection result; the Neck network downsamples the second fused 8*8 target detection result and performs feature fusion on the first 4*4 target detection result to obtain a first fused 4*4 target detection result; the Neck network obtains a second target detection result based on the first fused 4*4 target detection result, the second fused 8*8 target detection result, the second fused 16*16 target detection result and the first fused 32*32 target detection result; the Neck network sends the second target detection result to the output end.

[0064] In some optional embodiments, performing target detection on the received first feature map through the DETR network to obtain a first target detection result includes: receiving the first feature map through the DETR network, performing target detection on the first feature map, and obtaining a first target prediction box; screening the first target prediction box through the DETR network using weighted box fusion to generate a first detection box; and determining the first target detection result based on the first detection box through the DETR network.

[0065] Among them, Weighted Boxes Fusion (WBF) fully considers the role of each first target prediction box in the generation of the first detection box, that is, it assigns a weight to each first target prediction box according to its confidence, and obtains the first detection box according to the position and weight of each first target prediction box.

[0066] In some optional embodiments, the DETR network is used to filter the first target prediction frame using weighted frame fusion to generate a first detection frame, including: determining the weight of the first target prediction frame according to the confidence of the first target prediction frame; determining the first fusion confidence of the first fusion frame according to the first target prediction frame weight; if the first fusion confidence is greater than a first preset confidence threshold, determining the position coordinates of the upper left corner and lower right corner of the first fusion frame according to the upper left corner and lower right corner coordinate formulas; wherein the upper left corner coordinate formula is The formula for the coordinates of the lower right corner is Among them, c i is the confidence of the first target prediction box of the i-th object, (x1 i , y1 i ) is the coordinate of the upper left corner of the i-th first target prediction box, (x2 i , y2 i ) is the coordinate of the lower right corner of the i-th first target prediction box, (x1 m , y1 m ) is the upper left corner position coordinate of the mth first fusion frame, (x2 m , y2 m ) is the lower right corner position coordinate of the mth first fused frame; and the first detection frame is generated according to the upper left corner position coordinate and the lower right corner position coordinate of the first fused frame.

[0067] Among them, according to And the first target prediction box weight determines the first fusion confidence of the first fusion box; wherein, c m The first preset confidence threshold can be 0.9 or 0.85. The first preset confidence threshold can be adjusted according to specific actual conditions, and this application does not make any specific limitation on this.

[0068] In some optional embodiments, the step of performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result includes: receiving the first target detection result through the Neck network, performing feature fusion and feature extraction on the first target detection result to obtain a second target prediction frame; screening the second target prediction frame through the Neck network using weighted frame fusion to generate a second detection frame; and determining the second target detection result based on the second detection frame through the Neck network.

[0069] Among them, WBF fully considers the role of each second target prediction box in the generation of the detection box, that is, it assigns a weight to each second target prediction box according to the confidence level, and obtains the second detection box according to the position and weight of each second target prediction box.

[0070] In some optional embodiments, the second target prediction frame is screened by the Neck network using weighted frame fusion to generate a second detection frame, including: determining the weight of the second target prediction frame according to the confidence of the second target prediction frame; determining the second fusion confidence of the second fusion frame according to the second target prediction frame weight; if the second fusion confidence is greater than a second preset confidence threshold, determining the position coordinates of the upper left corner and lower right corner of the second fusion frame according to the upper left corner and lower right corner coordinate formula; wherein the upper left corner coordinate formula is The formula for the coordinates of the lower right corner is Among them, C i is the confidence of the i-th second target prediction box, (X1 i , Y1 i ) is the coordinate of the upper left corner of the i-th second target prediction box, (X2 i , Y2 i ) is the coordinate of the lower right corner of the i-th second target prediction box, (X1 m , Y1 m ) is the upper left corner position coordinate of the mth second fusion frame, (X2 m , Y2 m ) is the lower right corner position coordinate of the m-th second fused frame; and the second detection frame is generated according to the upper left corner position coordinate and the lower right corner position coordinate of the second fused frame.

[0071] Among them, according to And the second target prediction box weight determines the second fusion confidence of the second fusion box; wherein, C mThe second fusion confidence of the mth second fusion frame. The first preset confidence threshold and the second preset confidence threshold can be the same or different; the second preset confidence threshold can be 0.9 or 0.95. The second preset confidence threshold can be adjusted according to specific actual conditions and is not specifically limited in this application.

[0072] In some optional embodiments, the training process of the DETR-YOLO model includes: obtaining multiple training images and known targets in the training images; inputting the training images into the model to be trained to obtain the target detection results of the model to be trained; optimizing the internal parameters of the model to be trained according to the known targets and the target detection results to obtain a trained DETR-YOLO model.

[0073] Among them, training images can be obtained from images obtained by side-scan sonar equipment from actual measurements in my country's East China Sea, Yellow Sea, Bohai Sea and inland lakes.

[0074] Among them, before inputting the training image into the model to be trained, the training image can be normalized first, and the image data can be enhanced by using mosaic, image rotation, multi-scale cropping and magnification, image translation, image mirroring or image noise, thereby enriching the position distribution of the target and amplifying small-sized targets to a certain extent, thereby improving the generalization ability of the model while improving the model training efficiency.

[0075] Among them, the DETR-YOLO model can be built in Python based on the Pytorch framework, and the built DETR-YOLO model can be trained using a five-fold cross-calculation strategy.

[0076] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a DETR-YOLO-based target detection device provided in an embodiment of the present application, the DETR-YOLO-based target detection device includes:

[0077] Image acquisition module 201, the image acquisition module 201 is used to acquire the image to be detected;

[0078] An image input module 202 is used to input the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence;

[0079] The image detection module 203 is used to detect the target in the image to be detected according to the DETR-YOLO model.

[0080] In some optional embodiments, the DETR-YOLO-based target detection device may further include:

[0081] A training image acquisition module, configured to acquire a plurality of training images and known targets in the training images;

[0082] A target detection result acquisition module, wherein the target detection result acquisition module is used to input the training image into the model to be trained to obtain the target detection result of the model to be trained;

[0083] A model optimization module is used to optimize the internal parameters of the model to be trained according to the known target and the target detection result to obtain a trained DETR-YOLO model.

[0084] The implementation principle and technical effects of the target detection device based on DETR-YOLO provided in the embodiment of the present application have been introduced in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, please refer to the corresponding content in the method embodiment.

[0085] An embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are read and executed by a computer processor, the DETR-YOLO-based target detection method provided in the embodiment of the present application is executed.

[0086] In the embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0088] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0089] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A target detection method based on DETR-YOLO, characterized in that: The method comprises: Obtain the image to be detected; Inputting the image to be detected into a trained DETR-YOLO model; wherein the DETR-YOLO model includes a baseline network, a DETR network, and a Neck network connected in sequence; Detecting the target in the image to be detected according to the DETR-YOLO model; Detecting the target in the image to be detected according to the DETR-YOLO model includes: Performing feature extraction on the input image to be detected through the reference network to obtain a first feature map, and sending the first feature map to the DETR network; Performing target detection on the received first feature map through the DETR network to obtain a first target detection result, and sending the first target detection result to the Neck network; the DETR network includes an Encoder part, a Decoder part, and a Prediction part; Performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result; determining the target in the image to be detected according to the second target detection result; The performing target detection on the received first feature map through the DETR network to obtain a first target detection result includes: Performing position encoding on the first feature map by the Encoder part; The decoder part extracts features from the first feature map after position encoding output by the encoder part, and embeds a small number of fixed positions into object queries; Passing the output of the Decoder part to the feedforward neural network of the Prediction part; The feedforward neural network of the Prediction part performs network detection on the output of the Decoder part to obtain the first target detection result; wherein the first target detection result includes: the category and position of the target, or the no-target class.

2. The method according to claim 1, characterized in that The DETR-YOLO model also includes a SENet network, which is arranged between the baseline network and the DETR network; The step of extracting features of the input image to be detected by the reference network to obtain a first feature map, and sending the first feature map to the DETR network includes: Extract features of the input image to be detected through the reference network to obtain a second feature map, and send the second feature map to the SENet network; The second feature map is received through the SENet network, the second feature map is compressed and excited, a first feature map is obtained, and the first feature map is sent to the DETR network.

3. The method according to claim 1, characterized in that The DETR-YOLO model includes receptive fields of 4*4, 8*8, 16*16, and 32*32; The performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result includes: Receiving the first target detection result through the Neck network; wherein the first target detection result includes a first 4*4 target detection result, a first 8*8 target detection result, a first 16*16 target detection result, and a first 32*32 target detection result; The Neck network performs sampling and fusion operations on the first 4*4 target detection result, the first 8*8 target detection result, the first 16*16 target detection result, and the first 32*32 target detection result to obtain the second target detection result.

4. The method according to claim 1, wherein The performing target detection on the received first feature map through the DETR network to obtain a first target detection result includes: Receiving the first feature map through the DETR network, performing target detection on the first feature map, and obtaining a first target prediction box; The first target prediction frame is screened by the DETR network using weighted frame fusion to generate a first detection frame; Determine the first target detection result according to the first detection box through the DETR network.

5. The method according to claim 4, characterized in that The step of screening the first target prediction frame by using the weighted frame fusion through the DETR network to generate a first detection frame includes: Determining a weight of the first target prediction box according to a confidence level of the first target prediction box; Determine a first fusion confidence of the first fusion frame according to the weight of the first target prediction frame; If the first fusion confidence is greater than a first preset confidence threshold, the coordinates of the upper left corner and lower right corner of the first fusion frame are determined according to the upper left corner and lower right corner coordinate formulas; wherein the upper left corner coordinate formula is , the coordinate formula of the lower right corner is ;in, is the confidence of the i-th first target prediction box, is the coordinate of the upper left corner of the i-th first target prediction box, is the coordinate of the lower right corner of the i-th first target prediction box, is the upper left corner position coordinate of the mth first fusion frame, The coordinates of the lower right corner of the mth first fusion frame; The first detection frame is generated according to the upper left corner position coordinates and the lower right corner position coordinates of the first fused frame.

6. The method according to claim 1, characterized in that The performing feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result includes: receiving the first target detection result through the Neck network, performing feature fusion and feature extraction on the first target detection result, and obtaining a second target prediction frame; Using the Neck network to filter the second target prediction frame using weighted frame fusion to generate a second detection frame; Determine the second target detection result according to the second detection box through the Neck network.

7. The method according to claim 1, characterized in that The training process of the DETR-YOLO model includes: Acquire a plurality of training images and known targets in the training images; Inputting the training image into the model to be trained to obtain the target detection result of the model to be trained; The internal parameters of the model to be trained are optimized according to the known target and the target detection result to obtain a trained DETR-YOLO model.

8. A target detection device based on DETR-YOLO, characterized in that: The device comprises: An image acquisition module, wherein the image acquisition module is used to acquire an image to be detected; An image input module, wherein the image input module is used to input the image to be detected into the trained DETR-YOLO model; wherein the DETR-YOLO model includes an input module, a reference network, a DETR network, a Neck network, and an output module that are sequentially connected in communication; An image detection module is used to detect the target in the image to be detected according to the DETR-YOLO model; The image detection module is further configured to perform feature extraction on the input image to be detected through the reference network to obtain a first feature map, and send the first feature map to the DETR network; perform target detection on the received first feature map through the DETR network to obtain a first target detection result, and send the first target detection result to the Neck network; the DETR network includes an encoder part, a decoder part, and a prediction part; perform feature fusion and feature extraction on the received first target detection result through the Neck network to obtain a second target detection result; and determine the target in the image to be detected based on the second target detection result; The image detection module is specifically used to position encode the first feature map through the Encoder part; extract features from the first feature map after position encoding output by the Encoder part through the Decoder part, and embed a small fixed number of positions into Object Queries; pass the output of the Decoder part to the feedforward neural network of the Prediction part; the feedforward neural network of the Prediction part performs network detection on the output of the Decoder part to obtain the first target detection result; wherein, the first target detection result includes: the category and position of the target, or the no-target class.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, executes the method according to any one of claims 1 to 7.

Images