Small target object detection method and device, electronic equipment and readable storage medium

Abstract

The present disclosure provides a small target object detection method, which comprises: acquiring a detection image; performing down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain a multi-scale shallow feature image and a deep feature image; performing stripe feature extraction on the shallow feature image by using the small target detection model to obtain a stripe feature image; performing semantic feature extraction on the deep feature image by using the small target detection model to obtain a semantic feature image; and determining a small target object in the detection image according to the stripe feature image and the semantic feature image. The present disclosure also provides a small target object detection device, an electronic device and a readable storage medium. The present disclosure can improve the detection accuracy of small target objects and has a wider confidence threshold selection range without increasing the input resolution and the number of detection layers of the small target detection model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a small target object detection method and device, an electronic device, and a readable storage medium. BACKGROUND

[0002] With the continuous progress of science and technology, there are more and more types of target detection methods. With the help of target detection methods, automatic detection of target objects can be achieved.

[0003] In related technologies, the detection accuracy of small target detection models for small target objects is improved by increasing the input resolution of the small target detection model or increasing the number of detection layers of the small target detection model. However, this will increase the training time and inference time of the small target detection model.

[0004] Therefore, it is very important to improve the detection accuracy of small target objects without increasing the input resolution and the number of detection layers of the small target detection model. SUMMARY

[0005] To solve at least one of the above technical problems, the present disclosure provides a small target object detection method and device, an electronic device, and a readable storage medium.

[0006] The first aspect of the present disclosure provides a small target object detection method, comprising: acquiring a detection image; performing down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain a multi-scale shallow feature image and a deep feature image; performing stripe feature extraction on the shallow feature image by using the small target detection model to obtain a stripe feature image; performing semantic feature extraction on the deep feature image by using the small target detection model to obtain a semantic feature image; and determining a small target object in the detection image according to the stripe feature image and the semantic feature image.

[0007] In some embodiments, the small target detection model includes the backbone network, and the backbone network includes a first convolutional component and a second convolutional component. The down-sampling processing of the detection image in the backbone network by using the pre-trained small target detection model to obtain the multi-scale shallow feature image and the deep feature image includes: calling the first convolutional component to perform down-sampling processing on the detection image to obtain the shallow feature image; and calling the second convolutional component to perform down-sampling processing on the shallow feature image to obtain the deep feature image.

[0008] In some embodiments, the second convolution component includes a first convolution module, a first splicing module, and a plurality of second convolution modules with different parameters, and the calling the second convolution component to perform down-sampling processing on the shallow feature image to obtain the deep feature image includes: calling the plurality of second convolution modules to respectively perform convolution processing on the shallow feature image; calling the first splicing module to perform splicing processing on the plurality of convolution-processed shallow feature images to obtain a first spliced feature image; and calling the first convolution module to perform convolution processing on the first spliced feature image to obtain the deep feature image.

[0009] In some embodiments, the small target detection model includes a stripe feature extraction component, the stripe feature extraction component includes a first max-pooling module, a second max-pooling module, a third convolution module, and a second splicing module, and the adopting the small target detection model to perform stripe feature extraction on the shallow feature image to obtain a stripe feature image includes: calling the first max-pooling module and the second max-pooling module to respectively perform max-pooling processing on the shallow feature image to obtain a first feature image and a second feature image; calling the third convolution module to perform convolution processing on the second feature image; and calling the second splicing module to perform splicing processing on the first feature image and the convolution-processed second feature image to obtain the stripe feature image.

[0010] In some embodiments, the small target detection model includes a semantic feature extraction component, the semantic feature extraction component includes a spatial attention module, and the adopting the small target detection model to perform semantic feature extraction on the deep feature image to obtain a semantic feature image includes: calling the spatial attention module to perform small target object semantic feature positioning on the deep feature image to obtain the semantic feature image.

[0011] In some embodiments, the small target detection model includes an up-sampling component, a first multi-feature fusion component, and a first detection layer, and the determining the small target object in the detection image according to the stripe feature image and the semantic feature image includes: calling the up-sampling component to perform up-sampling combination processing on the deep feature image, so that the up-sampling combination-processed deep feature image has the same size as the stripe feature image; calling the first multi-feature fusion component to perform multi-feature fusion processing on the stripe feature image and the up-sampling combination-processed deep feature image to obtain a first multi-feature fusion image; and calling the first detection layer to perform small target object detection on the first multi-feature fusion image to determine the small target object in the detection image.

[0012] In some embodiments, the upsampling component includes a transpose convolution module, an upsampling module, and a third splicing module, and the upsampling combined processing of the deep feature image by the upsampling component includes: performing inverse convolution processing on the deep feature image by the transpose convolution module to obtain a third feature image; performing upsampling processing on the deep feature image by the upsampling module to obtain a fourth feature image; and performing splicing processing on the third feature image and the fourth feature image by the third splicing module to obtain the deep feature image after the upsampling combined processing.

[0013] In some embodiments, the small target detection model includes a third convolution component, a second multi-feature fusion component, and a second detection layer, and the determination of the small target object in the detection image according to the stripe feature image and the semantic feature image includes: performing downsampling processing on the stripe feature image by the third convolution component, so that the stripe feature image after the downsampling processing has the same size as the semantic feature image; performing multi-feature fusion processing on the semantic feature image and the stripe feature image after the downsampling processing by the second multi-feature fusion component to obtain a second multi-feature fusion image; and performing small target object detection on the second multi-feature fusion image by the second detection layer to determine the small target object in the detection image.

[0014] In some embodiments, before the pre-trained small target detection model is used to perform downsampling processing on the detection image in the backbone network to obtain multi-scale shallow feature images and deep feature images, the method further includes: constructing a first neural network model, a second neural network model, and a third neural network model, wherein the first neural network model includes a backbone network, a neck network, and a small target object detection layer, the second neural network model includes the backbone network, the neck network, and a negative sample detection layer, and the third neural network model includes the backbone network, the neck network, the small target object detection layer, and the negative sample detection layer; training the first neural network model by using a small target dataset to obtain a trained first neural network model; training the second neural network model by using a negative sample dataset to obtain a trained second neural network model; taking the backbone network weight, the neck network weight, and the small target object detection layer weight of the trained first neural network model as the backbone network initial weight, the neck network initial weight, and the small target object detection layer weight of the third neural network model; taking the negative sample detection layer weight of the trained second neural network model as the negative sample detection layer initial weight of the third neural network model; training the third neural network model by using the small target dataset to obtain a trained third neural network model; and taking the trained third neural network model as the small target detection model.

[0015] In some embodiments, the training of the third neural network model using the small target dataset comprises: freezing the weights of the negative sample detection layer of the third neural network model, and training the third neural network model using the small target dataset to obtain a to-be-corrected third neural network model; deleting the negative sample detection layer of the to-be-corrected third neural network model, and taking the to-be-corrected third neural network model after deleting the negative sample detection layer as the trained third neural network model.

[0016] In some embodiments, after the training of the first neural network model using the small target dataset to obtain a trained first neural network model, the method further comprises: performing small target object detection on the training images in the small target dataset using the trained first neural network model to obtain a small target object detection result; and generating the negative sample dataset according to the small target object detection result, wherein the negative sample is an object that is incorrectly identified as a small target object in the small target object detection result.

[0017] In some embodiments, before the downsampling processing of the detection image in the backbone network using the pre-trained small target detection model to obtain the multi-scale shallow feature image and the deep feature image, the method further comprises: constructing a first neural network model, wherein the first neural network model comprises a backbone network, a neck network, and a small target object detection layer; training the first neural network model using a small target dataset and a negative sample dataset to obtain a target first neural network model; and taking the target first neural network model as the small target detection model.

[0018] The second aspect of the present disclosure provides a small target object detection device, comprising: an acquisition module configured to acquire a detection image; a first processing module configured to perform downsampling processing of the detection image in a backbone network using a pre-trained small target detection model to obtain a multi-scale shallow feature image and a deep feature image; a second processing module configured to perform stripe feature extraction on the shallow feature image using the small target detection model to obtain a stripe feature image; a third processing module configured to perform semantic feature extraction on the deep feature image using the small target detection model to obtain a semantic feature image; and a determination module configured to determine a small target object in the detection image according to the stripe feature image and the semantic feature image.

[0019] The third aspect of the present disclosure provides an electronic device, comprising: a memory configured to store execution instructions; and a processor configured to execute the execution instructions stored in the memory, so that the processor performs the small target object detection method according to any one of the embodiments of the present disclosure.

[0020] The fourth aspect of the present disclosure provides a readable storage medium, wherein the readable storage medium stores execution instructions, and the execution instructions are executed by a processor to implement the small target object detection method in any of the embodiments of the present disclosure.

[0021] The small target object detection method, device, electronic equipment and readable storage medium provided by the present disclosure can use a pre-trained small target detection model to perform down-sampling processing on a detection image in a backbone network to obtain multi-scale shallow feature images and deep feature images, and then extract a stripe feature image of a small target object from the shallow feature images and extract a semantic feature image of the small target object from the deep feature images, so as to determine the small target object in the detection image by combining the stripe feature image and the semantic feature image. In this way, the detection accuracy of the small target object is improved without increasing the input resolution and the number of detection layers of the small target detection model, and the confidence threshold selection range is wider. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the general description of the disclosure given above, and the detailed description of the embodiments below, serve to explain the principles of the present disclosure. These drawings are included herewith and constitute a part of this specification.

[0023] Figure 1 is a flowchart of a small target object detection method of some embodiments of the present disclosure.

[0024] Figure 2 is a flowchart of obtaining shallow feature images and deep feature images by down-sampling processing in some embodiments of the present disclosure.

[0025] Figure 3 is a flowchart of obtaining deep feature images by a second convolution component in some embodiments of the present disclosure.

[0026] Figure 4 is a visual example diagram of a second convolution component in some embodiments of the present disclosure.

[0027] Figure 5 is a flowchart of obtaining stripe feature images by a stripe feature extraction component in some embodiments of the present disclosure.

[0028] Figure 6 is a visual example diagram of a stripe feature extraction component in some embodiments of the present disclosure.

[0029] Figure 7 is a pixel distribution example diagram of a shallow feature image in some embodiments of the present disclosure.

[0030] Figure 8is a maximum pooling result image obtained after maximum pooling of the shallow feature image with a stride of 1 and a pooling box of 3, according to some embodiments of the present disclosure.

[0031] Figure 9 is a maximum pooling result image obtained after maximum pooling of the shallow feature image with a stride of 2 and a pooling box of 3, according to some embodiments of the present disclosure.

[0032] Figure 10 is a flowchart of a process for determining a small target object by an upsampling component, a first multi-feature fusion component, and a first detection layer, according to some embodiments of the present disclosure.

[0033] Figure 11 is a flowchart of a process for upsampling and combining processing by an upsampling component, according to some embodiments of the present disclosure.

[0034] Figure 12 is a visual example diagram of an upsampling component, according to some embodiments of the present disclosure.

[0035] Figure 13 is a flowchart of a process for determining a small target object by a third convolution component, a second multi-feature fusion component, and a second detection layer, according to some embodiments of the present disclosure.

[0036] Figure 14 is a flowchart of a training process of a small target detection model, according to some embodiments of the present disclosure.

[0037] Figure 15 is a flowchart of a training process of a third neural network model, according to some embodiments of the present disclosure.

[0038] Figure 16 is a network structure diagram of a first neural network model, according to some embodiments of the present disclosure.

[0039] Figure 17 is a network structure diagram of a second neural network model, according to some embodiments of the present disclosure.

[0040] Figure 18 is a network structure diagram of a third neural network model, according to some embodiments of the present disclosure.

[0041] Figure 19 is a diagram illustrating module information in a network structure, according to some embodiments of the present disclosure.

[0042] Figure 20 is a comparison diagram of F1-Confidence curves of a small target detection model and a Yolov5s model in a related art before improvement, according to some embodiments of the present disclosure.

[0043] Figure 21 ,Figure 22 and Figure 23 is a comparison chart of detection effects of a small target detection model of some embodiments of the present disclosure and a Yolov5s model in prior related technologies.

[0044] Figure 24 is another training flowchart of a small target detection model of some embodiments of the present disclosure.

[0045] Figure 25 is a structural schematic block diagram of a small target object detection device using a hardware implementation of a processing system of one embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The present disclosure will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only intended to explain the related content, and not to limit the present disclosure. In addition, it should be noted that only parts related to the present disclosure are shown in the drawings for ease of description.

[0047] It should be noted that the embodiments and features in the embodiments in the present disclosure can be combined with each other without conflict. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments.

[0048] Unless otherwise specified, the exemplary embodiments / instances shown will be understood to provide exemplary features of various details that can implement the technical concepts of the present disclosure in practice. Therefore, unless otherwise specified, the features of various embodiments / instances can be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of the present disclosure.

[0049] In the drawings, cross-hatching and / or shading are generally used to make the boundaries of adjacent components clear. Thus, the presence or absence of cross-hatching or shading does not convey or imply any preference or requirement as to the specific material, material properties, dimensions, proportions, commonality of the illustrated components, and / or any other characteristic, attribute, property, etc. of the components. In addition, in the drawings, the size and relative sizes of components can be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, a specific process sequence can be performed in a different order than described. For example, two consecutively described processes can be performed substantially simultaneously or in an order opposite to that described. In addition, the same reference numbers represent the same components.

[0050] When a component is referred to as being "on" or "on top of", "connected to", or "coupled to" another component, it can be directly on, directly connected to, or directly coupled to the other component, or intervening components can be present. However, when a component is referred to as being "directly on", "directly connected to", or "directly coupled to" another component, there are no intervening components present. For this reason, the term "connected" can refer to a physical or electrical connection, with or without intervening components.

[0051] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms "including", "includes", "having", "has", "a", "an", "one" or "said" and variants thereof are used in either the detailed description and / or the claims, such terms are intended to be inclusive in a manner similar to the term "comprising" or "including" as an open transition term without precluding any additional or other elements.

[0052] The following detailed description is presented in connection with Figures 1 to 23 The small target object detection method and small target object detection device of the present disclosure are described in detail.

[0053] Figure 1 is a flowchart of the small target object detection method of some embodiments of the present disclosure. The execution subject of the small target object detection method of the present disclosure can be an electronic device of the present disclosure or a small target object detection device integrated in the electronic device, etc. Please refer to Figure 1 The small target object detection method S100 provided by the present disclosure includes the following steps S110, S120, S130, S140 and S150.

[0054] S110: Obtain a detection image.

[0055] The small target object detection method of the present disclosure can be applied to an airport runway object detection scene, a ground target detection scene in aerial photography, or other scenes that need to detect objects at a long distance, etc.

[0056] Exemplarily, in the airport runway object detection scene, the detection image can be a photographed airport runway image; in the ground target detection scene in aerial photography, the detection image can be an aerial photography ground image.

[0057] S120: Down-sampling the detection image in the backbone network of the pre-trained small target detection model to obtain multi-scale shallow feature images and deep feature images.

[0058] Exemplarily, the small target detection model can be obtained by improving the Yolov5s model in the related art.

[0059] It can be understood that in the backbone network of the small target detection model, multi-scale shallow feature images and deep feature images can be generated by down-sampling the detection image. The scale of the detection image is larger than that of the shallow feature image. The scale of the shallow feature image is larger than that of the deep feature image.

[0060] It should be noted that the number of shallow feature images can be multiple, and the number of deep feature images can also be multiple, and the number of shallow feature images and the number of deep feature images are not limited here.

[0061] S130: Extracting stripe features from the shallow feature image using the small target detection model to obtain a stripe feature image.

[0062] Exemplarily, the small target detection model can generate a stripe feature image by extracting stripe features from the shallow feature image.

[0063] S140: Extracting semantic features from the deep feature image using the small target detection model to obtain a semantic feature image.

[0064] Exemplarily, the small target detection model can generate a semantic feature image by extracting semantic features from the deep feature image.

[0065] S150: Determining a small target object in the detection image according to the stripe feature image and the semantic feature image.

[0066] Exemplarily, the stripe feature image mainly includes stripe features but also includes part of semantic features. The semantic feature image mainly includes semantic features but also includes part of stripe features. The stripe feature image is mainly used to let the small target detection model determine that there is a shape similar to the small target object we need in a certain place, that is, to better locate the position where the small target object may appear. The semantic feature image is mainly used to determine the probability that this small target object is the small target object we need.

[0067] Exemplarily, the small target object can be understood as a target object whose area ratio in the image to the total area of the image is less than 0.03.

[0068] The small target object detection method provided by the present disclosure can use a pre-trained small target detection model to perform down-sampling processing on a detection image in a backbone network to obtain multi-scale shallow feature images and deep feature images, and then extract a stripe feature image of a small target object from the shallow feature images and extract a semantic feature image of the small target object from the deep feature images, so as to determine the small target object in the detection image in combination with the stripe feature image and the semantic feature image. In this way, the detection accuracy of the small target object is improved and the confidence threshold selection range is wider without increasing the input resolution and the number of detection layers of the small target detection model.

[0069] In some embodiments, the small target detection model comprises a backbone network, and the backbone network comprises a first convolution component and a second convolution component. Please refer to Figure 2 , accordingly, step S120 comprises:

[0070] S121: calling the first convolution component to perform down-sampling processing on the detection image to obtain a shallow feature image.

[0071] S122: calling the second convolution component to perform down-sampling processing on the shallow feature image to obtain a deep feature image.

[0072] Specifically, the parameters of the first convolution component are different from the parameters of the second convolution component. Calling the first convolution component to perform down-sampling processing on the detection image can obtain a shallow feature image with a smaller scale than the detection image. Calling the second convolution component to perform down-sampling processing on the shallow feature image can obtain a deep feature image with a smaller scale than the shallow feature image.

[0073] In some embodiments, the second convolution component comprises a first convolution module, a first splicing module, and a plurality of second convolution modules with different parameters. Please refer to Figure 3 , accordingly, step S122 comprises:

[0074] S1221: calling the plurality of second convolution modules to respectively perform convolution processing on the shallow feature image;

[0075] S1222: calling the first splicing module to perform splicing processing on the plurality of convolution-processed shallow feature images to obtain a first spliced feature image;

[0076] S1223: calling the first convolution module to perform convolution processing on the first spliced feature image to obtain a deep feature image.

[0077] In this way, while performing down-sampling processing on the shallow feature image, the receptive field can be expanded and multi-scale context information can be captured, so that the semantic features of the small target object can be better extracted.

[0078] In one example, the first convolution component includes a convolution (Conv) module of a Yolov5s model in the related art, and the second convolution component includes a first convolution module, a first concatenation (Concat) module, and three second convolution modules with different dilation rates. The first convolution module is a Conv module of the Yolov5s model in the related art, the second convolution module is a dilated convolution (Dilated Convolution) module, and the three dilated convolution modules are arranged in parallel. The dilation rates of the three dilated convolution modules are 1, 2, and 3, respectively, as shown in Figure 4 Further, the shallow feature image is input into the three second convolution modules, the three second convolution modules are used to perform convolution processing on the shallow feature image, then the three shallow feature images after convolution processing are input into the first concatenation module, the first concatenation module is used to perform concatenation processing on the three shallow feature images after convolution processing and output a first concatenation feature image, then the first concatenation feature image is input into the first convolution module, the first convolution module is used to perform convolution processing on the first concatenation feature image and output a deep feature image. It is worth noting that in this example, the second convolution component is not used to perform down-sampling processing on the detection image to obtain the shallow feature image because the dilation rate will cause the convolution kernel to be hollow, and the hollow part will not participate in the calculation when extracting features, which will cause some stripe features to be lost, thereby reducing the quality of the stripe feature image obtained from the shallow feature image.

[0079] In another example, the first convolution component includes a Conv module of a Yolov5s model in the related art, and the second convolution component includes a first convolution module, a first concatenation module, and three second convolution modules with different convolution kernels. The first convolution module and the second convolution module are both ordinary convolution modules, and the three second convolution modules are arranged in parallel.

[0080] In some embodiments, the small target detection model includes a stripe feature extraction component, and the stripe feature extraction component includes a first max-pooling module, a second max-pooling module, a third convolution module, and a second concatenation module. Please refer to Figure 5 , accordingly, step S130 includes:

[0081] S131: calling the first max-pooling module and the second max-pooling module to perform max-pooling processing on the shallow feature image to obtain a first feature image and a second feature image.

[0082] S132: calling the third convolution module to perform convolution processing on the second feature image.

[0083] S133: calling the second splicing module to splice the first feature image and the second feature image processed by the convolution to obtain the stripe feature image.

[0084] Thus, in the down-sampling process, the second max pooling with unchanged input and output dimensions is used first, and then the third convolution module is used for down-sampling, so that more stripe features are retained and the stripe features are increased, and the quality of the obtained stripe feature image is improved.

[0085] For example, please refer to Figure 7 , Figure 8 and Figure 9 , wherein, Figure 7 is an example of pixel distribution of a shallow feature image, Figure 8 is a max pooling result obtained by performing max pooling on the shallow feature image with a stride of 1 and a pooling box of 3, Figure 9 is a max pooling result obtained by performing max pooling on the shallow feature image with a stride of 2 and a pooling box of 3, wherein the number 1 in the figure represents the background, and the numbers other than 1 represent stripe features. Figure 7 and Figure 8 , it can be found that the size of the feature image processed by the max pooling with a stride of 1 is unchanged, and the black stripe features in the processed feature image are enhanced; by comparing Figure 7 and Figure 9 , it can be found that the black stripe features in the feature image processed by the max pooling with a stride of 2 will be reduced, and the stripe features will be lost. In the stripe feature extraction assembly shown in Figure 6 , the second max pooling module with a stride of 1 is used to expand the stripe features in the shallow feature image to obtain the first feature image, and then the third convolution module is used to down-sample the first feature image, and then the first feature image processed by the third convolution module and the second feature image obtained by directly using the first max pooling module with a stride of 2 for down-sampling are input into the second splicing module, so that more stripe features are retained, and the defect that the stripe features are lost when the shallow feature image is directly down-sampled by the max pooling with a stride of 2 can be compensated.

[0086] It can be understood that if the shallow feature image is directly down-sampled by a max pooling module to obtain a stripe feature image, the stripe features will be lost, and the quality of the obtained stripe feature image will be reduced.

[0087] In one example, the shallow feature image is a 1 / 2 feature image, the stride (abbreviated as str) of the first max pooling module is 2, the stride of the second max pooling module is 1, and the stride of the third convolution module is 2. The second max pooling module and the third convolution module are connected in series, and then connected in parallel with the first max pooling module, as shown inFigure 6 As shown in FIG. 6, the stripe feature extraction component includes a first max pooling module, a second max pooling module, and a third convolution module. The first max pooling module is connected in series with the second max pooling module, and the second max pooling module is connected in series with the third convolution module. The first max pooling module, the second max pooling module, and the third convolution module are connected in parallel with the first max pooling module. In this way, the stripe feature extraction component, on the one hand, down-samples the 1 / 2 feature image, and on the other hand, increases the stripe features of the 1 / 2 feature image and retains more stripe features.

[0088] In another example, the shallow feature image is a 1 / 4 feature image, the stride of the first max pooling module is 1, the stride of the second max pooling module is 1, and the stride of the third convolution module is 1. The second max pooling module and the third convolution module are connected in series, and the series connection of the second max pooling module and the third convolution module is connected in parallel with the first max pooling module. In this way, the stripe feature extraction component increases the stripe features of the 1 / 4 feature image and retains more stripe features.

[0089] It is worth noting that Max Pool only extracts the maximum value of the feature value in the shallow feature image, and does not calculate a new value from the feature value and the parameter, so that the original stripe features in the shallow feature image can be retained.

[0090] In some embodiments, the small target detection model includes a semantic feature extraction component, and the semantic feature extraction component includes a space attention module. Accordingly, step S140 includes:

[0091] S141: calling the space attention module to perform small target object semantic feature positioning on the deep feature image to obtain a semantic feature image.

[0092] In this way, using the space attention mechanism to position the small target object in the deep feature image can enable the small target detection model to better and more accurately pay attention to the semantic features at the position of the small target object, thereby improving the quality of the obtained semantic feature image.

[0093] It can be understood that the space attention is used instead of the channel attention because the positioning of the small target object in the deep feature image is more difficult than the feature extraction, and when the positioning is wrong, the corresponding feature extraction is also wrong.

[0094] In some embodiments, the small target detection model includes an up-sampling component, a first multi-feature fusion component, and a first detection layer. Please refer to FIG. 7, and accordingly, step S150 includes: Figure 10

[0095] S151: calling the up-sampling component to perform up-sampling combination processing on the deep feature image, so that the deep feature image after the up-sampling combination processing has the same size as the stripe feature image.

[0096] S152: calling the first multi-feature fusion component to perform multi-feature fusion processing on the stripe feature image and the deep feature image after the up-sampling combination processing to obtain a first multi-feature fusion image.​​

[0097] S153: calling the first detection layer to perform small target object detection on the first multi-feature fusion image to determine the small target object in the detection image.

[0098] In this way, the small target object with a smaller scale in the detection image can be determined.

[0099] Exemplarily, the first multi-feature fusion component can include a fourth splicing module, a fourth convolution module and a first multi-feature layer fusion module connected in sequence, wherein the fourth splicing module is the original Concat module in the Yolov5s model of the related technology, the fourth convolution module is the original Conv module in the Yolov5s model of the related technology, and the first multi-feature layer fusion module is the original C3 module in the Yolov5s model of the related technology. Then in step S152, the stripe feature image and the deep layer feature image after the combination processing of upsampling are input into the fourth splicing module, the stripe feature image and the deep layer feature image after the combination processing of upsampling are spliced by the fourth splicing module to obtain a second spliced feature image, then the second spliced feature image is input into the fourth convolution module, the second spliced feature image is feature extracted by the fourth convolution module, and finally the second spliced feature image after the feature extraction is input into the first multi-feature layer fusion module, and the image features are further fused by the first multi-feature layer fusion module to obtain the first multi-feature fusion image.

[0100] Further, the small target detection model further includes a second upsampling component and a second multi-feature layer fusion module, the second upsampling component has the same composition as the upsampling component in step S151, and the second multi-feature layer fusion module is the original C3 module in the Yolov5s model of the related technology. Then in step S153, the first multi-feature fusion image is input into the second upsampling component for upsampling, the first multi-feature fusion image after the upsampling is input into the second multi-feature layer fusion module for image feature fusion, and finally the fused image is input into the small target object detection layer for small target object detection to determine the small target object in the detection image.

[0101] In some embodiments, the upsampling component includes a transpose convolution module, an upsampling module and a third splicing module. Please refer to Figure 11 , accordingly, step S151, comprising:

[0102] S1511: calling the transpose convolution module to perform inverse convolution processing on the deep layer feature image to obtain a third feature image.

[0103] S1512: calling the upsampling module to perform upsampling processing on the deep layer feature image to obtain a fourth feature image.

[0104] S1513: calling the third splicing module to perform splicing processing on the third feature image and the fourth feature image to obtain a deep feature image after up-sampling and combination processing.

[0105] Exemplarily, the up-sampling module is an Upsampling module in a Yolov5s model of the related art, and the third splicing module is a Concat module in the Yolov5s model of the related art.

[0106] It can be understood that the pixels of a small target object in an image are few, and the features of the small target object will gradually disappear in a deep network through continuous down-sampling convolution calculation of a backbone network. At this time, if only the Upsampling module in the Yolov5s model of the related art is used to perform up-sampling on a deep feature image in the deep network, since the Upsampling module is used to expand the size of an upper layer feature image by copying the feature values of the upper layer, and the features of the small target object in the deep feature image are too few or even disappear, in this case, the Upsampling module will only copy the feature values that are not obvious or wrong in the deep network, which cannot reproduce or expand the features of the small target object in the deep network, and even will expand the wrong features, thereby increasing the difficulty of a small target detection model in identifying the small target object.

[0107] However, in the embodiments of the present disclosure, an up-sampling assembly is adopted instead of the Upsampling module in the Yolov5s model of the related art. Please refer to Figure 12 The up-sampling assembly includes a Transposed Convolution module and an Upsampling module, and the Transposed Convolution module and the Upsampling module are further connected in parallel through the third splicing module (i.e., the Concat module in the figure). Since the Transposed Convolution module is used to calculate the feature values needed for expansion by parameters based on the feature values of the deep network, so as to expand the feature image, this expansion method can use the calculation of the parameters to derive the feature values of the correct learning direction from the feature values calculated incorrectly in the deep network, and even can strengthen or expand the features of the small target object. Therefore, the up-sampling assembly of the present disclosure is used to perform up-sampling and combination processing on the deep feature image, which can obtain the features of the deep network in the deep feature image and optimize and remove the wrong features.

[0108] It should be noted that compared with using the Transposed Convolution module alone for upsampling and using the Upsampling module alone for upsampling, many incorrect results appear in the results obtained by using the Transposed Convolution module, which do not appear in the results obtained by using the Upsampling module. This is because the Transposed Convolution module is calculated by parameters and deep feature values to obtain expanded feature values, which is a correct choice for the region where small target objects in the deep feature image are located, but for the background region, some smaller features similar to the small target objects will be mined out, so the background region needs the Upsampling module to copy the feature value method more. Therefore, the upsampling assembly of the present disclosure connects the Upsampling module and the Transposed Convolution module in parallel, and the method of connecting the background feature image obtained by the Upsampling module and the background feature image obtained by the Transposed Convolution module in parallel, fuses the correct background features obtained by the Upsampling module and the incorrect background features obtained by the Transposed Convolution module, and makes the incorrect learning direction of the background features learn to shift to the correct background feature direction.

[0109] In some embodiments, the small target detection model includes a third convolution assembly, a second multi-feature fusion assembly, and a second detection layer. Please refer to Figure 13 , accordingly, step S150 includes:

[0110] S154: calling the third convolution assembly to perform down-sampling processing on the stripe feature image, so that the down-sampled stripe feature image has the same size as the semantic feature image.

[0111] S155: calling the second multi-feature fusion assembly to perform multi-feature fusion processing on the semantic feature image and the down-sampled stripe feature image to obtain a second multi-feature fusion image.

[0112] S156: calling the second detection layer to perform small target object detection on the second multi-feature fusion image to determine the small target objects in the detection image.

[0113] In this way, the small target objects with larger scales in the detection image can be determined.

[0114] Exemplarily, the third convolution module is a Conv module in a Yolov5s model of the related art, and the second detection layer is a middle target detection layer in the Yolov5s model of the related art, which can detect small target objects with a larger scale in a detection image. The second feature fusion assembly can include a fifth splicing module, a fifth convolution module and a third multi-feature layer fusion module connected in sequence, wherein the fifth splicing module is a Concat module in the Yolov5s model of the related art, the fifth convolution module is a Conv module in the Yolov5s model of the related art, and the third multi-feature layer fusion module is a C3 module in the Yolov5s model of the related art. The implementation manner of step S155 is similar to that of the foregoing step S152, and thus details are not repeated herein.

[0115] Referring to Figure 14 In some embodiments, before step S120, the method further includes:

[0116] S210: constructing a first neural network model, a second neural network model and a third neural network model, wherein the first neural network model includes a backbone network, a neck network and a small target object detection layer, the second neural network model includes the backbone network, the neck network and a negative sample detection layer, and the third neural network model includes the backbone network, the neck network, the small target object detection layer and the negative sample detection layer.

[0117] S220: training the first neural network model by using a small target data set to obtain a trained first neural network model.

[0118] S230: training the second neural network model by using a negative sample data set to obtain a trained second neural network model.

[0119] S240: taking the backbone network weight, the neck network weight and the small target object detection layer weight of the trained first neural network model as the backbone network initial weight, the neck network initial weight and the small target object detection layer weight of the third neural network model.

[0120] S250: taking the negative sample detection layer weight of the trained second neural network model as the negative sample detection layer initial weight of the third neural network model.

[0121] S260: training the third neural network model by using the small target data set to obtain a trained third neural network model.

[0122] S270: taking the trained third neural network model as a small target detection model.

[0123] Thus, by means of contrastive learning, a small target detection model is trained based on the small target data set and the negative sample data set. Even if the number of samples in the negative sample data set is small, a small target detection model with high detection accuracy can be successfully trained.

[0124] It can be understood that when the small target object is small enough, the similarity between the features of the small target object and the features of the negative sample is high, the first neural network model after training cannot eliminate similar features, and the negative sample will be incorrectly detected as a small target object, while the second neural network model after training can effectively identify the common features of the negative sample. Therefore, by using the contrastive learning method of the present embodiment, the backbone network weight, the neck network weight and the small target object detection layer weight of the first neural network model after training are used as the initial weight of the backbone network, the initial weight of the neck network and the weight of the small target object detection layer of the third neural network model, and the negative sample detection layer weight of the second neural network model after training is used as the initial weight of the negative sample detection layer of the third neural network model. Then, the third neural network model is trained to obtain the third neural network model after training, which can effectively eliminate the features of the negative sample and the small target object, and make the third neural network model after training more focused on extracting other features of the small target object different from the negative sample.

[0125] For example, the third neural network model can be obtained by retaining the small target object detection layer in the Yolov5s model of the related art, deleting the large target object detection layer in the Yolov5s model of the related art, and connecting a negative sample detection layer identical to the small target object detection layer in parallel. In this way, without increasing the number of detection layers, a small target detection model with better small target object detection capability can be trained using a small target data set. The first neural network model can be understood as a model obtained by deleting the negative sample detection layer in the third neural network model. The second neural network model can be understood as a model obtained by deleting the small target object detection layer in the third neural network model. For the network structures in the first neural network model and the second neural network model that are identical to those in the Yolov5s model of the related art, the initial weights before training can directly use the pre-training weights of the corresponding network structures in the Yolov5s model of the related art. For the network structures in the first neural network model and the second neural network model that are different from those in the Yolov5s model of the related art, the initial weights before training can directly use the default weights set in advance.

[0126] For example, the small target data set can be obtained online. The small target data set can include multiple images labeled with small target objects. The negative sample data set can include multiple images labeled with negative samples.

[0127] Please refer to Figure 15In some embodiments, step S260 comprises:

[0128] S261: freezing the weights of the negative sample detection layer of the third neural network model, and training the third neural network model using the small target data set to obtain a to-be-corrected third neural network model.

[0129] S262: deleting the negative sample detection layer of the to-be-corrected third neural network model, and taking the to-be-corrected third neural network model after deleting the negative sample detection layer as a trained third neural network model.

[0130] In this way, the trained third neural network model can effectively eliminate the features of the negative samples that are the same as those of the small target objects, so that the trained third neural network model focuses more on extracting other features of the small target objects that are different from the negative samples, so that the trained third neural network model avoids incorrectly identifying non-small target objects as small target objects in actual detection, and improves the detection accuracy of the trained third neural network model.

[0131] Specifically, in step S261, the network adjusts the weights of each network structure of the third neural network model through learning so that the loss value of the negative sample detection layer tends to 1, the purpose being to let the third neural network model not detect the negative sample as a target, meaning that the backbone network and the neck network of the third neural network model do not calculate the feature values of the negative sample, and at the same time, the loss value calculated by the small target object class detection result tends to 0, the purpose being to let the third neural network model detect the small target object as a target, and in summary, this is to better eliminate the similar negative sample features contained in the small target features extracted by the network.

[0132] In one example, Figure 16 The network structure of the first neural network model is shown, Figure 17 The network structure of the second neural network model is shown, Figure 18 The network structure of the third neural network model is shown, wherein the upper left corner of each module is the module name, the upper right corner is the number of layers in the network structure, and the following is the module parameters, such as Figure 19The first neural network model, the second neural network model and the third neural network model shown in the figure are improved on the basis of the Yolov5s model in the related art, wherein Max-Copy can be understood as the stripe feature extraction component in the foregoing, Rateunion can be understood as the second convolution component in the foregoing, Space Attention can be understood as the semantic feature extraction component in the foregoing, Sample-Transpose can be understood as the up-sampling component in the foregoing, negative sample Detect can be understood as the negative sample detection layer in the foregoing, small target Detect can be understood as the first detection layer in the foregoing, medium target Detect can be understood as the second detection layer in the foregoing, and Conv, C3 and Sppf can be understood as the original modules of the Yolov5s model in the related art. Figure 16 In the example of Figure 18 , the Sample-Transpose at the 32nd layer, the C3 at the 33rd layer and the small target Detect can be understood as the small target object detection layer in the foregoing. Figure 17 In the example of Figure 18 , the Sample-Transpose at the 32nd layer, the C3 at the 33rd layer and the negative sample Detect can be understood as the negative sample detection layer in the foregoing.

[0133] The first neural network model, the second neural network model and the third neural network model of the present disclosure are the backbone network from the 0th layer to the 10th layer, wherein the 1st layer outputs a 1 / 2 feature image, the 3rd layer outputs a 1 / 4 feature image, the 5th layer outputs a 1 / 8 feature image, the 7th layer outputs a 1 / 16 feature image, and the 10th layer outputs a 1 / 32 feature image. The 1 / 2 feature image, the 1 / 4 feature image and the 1 / 8 feature image can be understood as shallow layer feature images, and the 1 / 16 feature image and the 1 / 32 feature image can be understood as deep layer feature images. The Yolov5s model in the related art is mainly to extract semantic features on the 1 / 8 feature image, the 1 / 16 feature image and the 1 / 32 feature image (similar to the network part of the connection arrow downward from the 5th, 7th and 10th layers in Figure 16 ), while the first neural network model, the second neural network model and the third neural network model of the present disclosure are mainly to extract and retain stripe features by adding feature extraction on the 1 / 4 feature image and the 1 / 2 feature image (such as the network part of the connection arrow downward from the 1st and 3rd layers in Figure 16 ), that is, the first neural network model, the second neural network model and the third neural network model of the present disclosure add the extraction of stripe features in the shallow layer backbone network.

[0134] Further, the Yolov5s model in the related art directly uses a Conv component for down-sampling at layers 4, 6, and 8. The first neural network model, the second neural network model, and the third neural network model of the present disclosure replace the Conv component in the Yolov5s model in the related art with the second convolution component described above for down-sampling at layers 4, 6, and 8, thereby better extracting semantic features of small target objects.

[0135] Next, the Yolov5s model in the related art detects small target objects, medium target objects, and large target objects at positions of 1 / 8 feature images, 1 / 16 feature images, and 1 / 32 feature images, respectively. According to the scale of a feature image input to a current Detect layer, it can be determined at which position of a feature image the current Detect layer is located. For example, if the scale of the feature image input to the current Detect layer is 1 / 8 of the detection image, the current Detect layer is located at the position of the 1 / 8 feature image. Since the features of small target objects almost disappear at the position of the 1 / 32 feature image, and the present disclosure focuses on the detection of small target objects, the large target Detect layer of the Yolov5s model in the related art at the position of the 1 / 32 feature image is deleted, the medium target Detect layer at the position of the 1 / 16 feature image is retained, and the upper layer feature image of the small target Detect layer at the position of the 1 / 8 feature image is connected to the small target Detect layer after being up-sampled once by the Sample-Transpose component, thereby optimizing the small target Detect layer at the position of the 1 / 8 feature image to the position of the 1 / 4 feature image.

[0136] The reason for modifying the Detect layer at the position of the 1 / 8 feature image to the position of the 1 / 4 feature image is that the features of small target objects do not fully appear at the position of the 1 / 8 feature image, and the depth of learning for the stripe features extracted by the shallow layer is not enough, and needs to be detected after several convolution calculations without changing the size of the image. No other layers are added between layers 30 and 33 for feature fusion, in order to avoid the influence of incorrect features of the deep layer and insufficiently learned features of the shallow layer on the final correct learning result of the first neural network model. Detecting the medium-to-large target objects at the position of the 1 / 16 feature image is less likely to lose features than the target objects detected by the small target Detect layer, and is in a deep network, so the semantic features are rich, and the Sample-Transpose component is not used for up-sampling again.

[0137] It should be noted that since the third neural network model (i.e., the small target detection model) after training is obtained by deleting the negative sample detection layer on the basis of the third neural network model, the network structure of the small target detection model is the same as that of the first neural network model, that is, the network structure of the small target detection model is also as shown in Figure 16 .

[0138] It can be understood that the F1 value is a technical index for comprehensively evaluating the detection result of the model, and the calculation formula thereof is: F1=(2*Precision*Recall) / (Precision+Recall), wherein Precision represents the precision, Recall represents the recall, and Confidence represents the confidence of the target. The F1-Confidence curve represents the change curve of the F1 value at different Confidences. The Map value is the area surrounded by the Percision-Recall image curve and the coordinate axis at different Confidence values, and the closer the Map value is to 1, the better the model is. Therefore, the performance of the small target detection model of the present disclosure and the Yolov5s model in the related art before improvement can be compared based on the F1-Confidence curve and the Map value.

[0139] The comparison of the F1-Confidence curves of the small target detection model of the present disclosure and the Yolov5s model in the related art before improvement is shown in Figure 20 . It can be seen that the F1 value in the F1-Confidence curve of the small target detection model of the present disclosure can reach a higher peak value and the F1 peak value changes smoothly in a wider Confidence threshold range without increasing the input resolution of the model and the number of detection layers, the generalization ability of the small target detection model of the present disclosure is stronger, and the memory occupied and the time consumed by inference of the model due to the increase of the input resolution and the number of detection layers can be reduced.

[0140] The comparison data of the Map values of the small target detection model of the present disclosure and the Yolov5s model in the related art before improvement is shown in Table 1. It can be seen that the Map value of the small target detection model of the present disclosure is greatly improved.

[0141] Table 1

[0142] Model Percision Recall Map@.5 Map@.5:.95 Related technology Yolov5s model 0.5508 0.39344 0.37783 0.16365 Small target detection model of the present disclosure 0.76256 0.61572 0.69696 0.36668

[0143] The comparison of the detection effects of the small target detection model of the present disclosure and the Yolov5s model in the related art before improvement on the same test set is shown in Figure 21 , Figure 22 and Figure 23The yellow box is a false small target object, and the green box is a recalled small target object. It can be seen that the confidence of the small target detection model of the present disclosure for small target object detection is significantly improved, and the precision and recall rate are also improved. In summary, the small target object detection method of the present disclosure improves the technical indicators of small target object detection.

[0144] In some embodiments, after step S260, further comprising: adding a large target object detection layer in the trained third neural network model to obtain a multi-target detection model; training the multi-target detection model using a preset data set labeled with large target objects, medium target objects and small target objects to obtain a trained multi-target detection model, the trained multi-target detection model being capable of detecting large target objects, medium target objects and small target objects. The large target object detection layer can be the large target object detection layer in the aforementioned deleted related technology Yolov5s model. Because the feature extraction of large target objects is easy, the aforementioned deleted large target object detection layer is restored, and large target objects are added to the data set, the training result of the small target detection model is continued, and the small target detection model with the added large target object detection layer is further learned, so that the similar features of the most difficult to learn small target objects and negative samples are removed, and the common features of large target objects, medium target objects and small target objects are better extracted.

[0145] Please refer to Figure 24 In some embodiments, before step S120, further comprising:

[0146] S310: constructing a first neural network model, the first neural network model comprising a backbone network, a neck network and a small target object detection layer.

[0147] S320: training the first neural network model using a small target data set and a negative sample data set to obtain a target first neural network model.

[0148] S330: taking the target first neural network model as a small target detection model.

[0149] Exemplarily, the network structure of the first neural network model is as shown in Figure 16

[0150] It can be understood that, since the first neural network model does not add a negative sample detection layer, the first neural network model can be directly trained using a data set labeled with small target objects and negative samples, without the need for training in the comparison learning manner of steps S210 to S270. Although this requires a larger number of negative sample data sets than the comparison learning manner, it requires less code than the comparison learning manner, thereby improving the convenience of model training.

[0151] ​In one example, in step S320, the negative sample dataset is first added to the small target dataset to form a new dataset, the new dataset only labels the small target object and does not label the negative sample, and then the new dataset is used to train the first neural network model, so as to obtain the target first neural network model. It can be understood that although the new dataset contains negative sample images, they are not labeled as small target objects or negative samples. Therefore, when the weights are modified by back propagation during the training of the first neural network model, the negative samples will be modified in the direction of not being identified as small target objects again, so as to obtain a small target detection model with high detection accuracy for small target objects.

[0152] It should be noted that in some embodiments, the negative sample dataset of the present disclosure can be obtained in the following manner: the first neural network model is trained using the small target dataset to obtain a trained first neural network model; the trained first neural network model is used to detect small target objects in the training images in the small target dataset to obtain small target object detection results; and the negative sample dataset is generated according to the small target object detection results, and the negative sample is an object that is incorrectly identified as a small target object in the small target object detection results.

[0153] In this way, the accuracy of the negative sample dataset is ensured.

[0154] For example, the small target object detection results obtained by the trained first neural network model include correctly identified small target objects and objects incorrectly identified as small target objects, the objects incorrectly identified as small target objects are used as negative samples, and more images containing the same kind of negative samples are obtained by crawling pictures, and then a negative sample dataset is generated using all the obtained negative sample images. In the negative sample dataset, the objects incorrectly identified as small target objects by the trained first neural network model are used as the labeling target.

[0155] Figure 25 is a structural schematic block diagram of a small target object detection device of one embodiment of the present disclosure, which is a hardware implementation of a processing system.

[0156] The device can include corresponding modules that perform one or more steps in the above flowchart. Therefore, each step or several steps in the above flowchart can be performed by corresponding modules, and the device can include one or more of these modules. The modules can be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer readable medium for implementation by a processor, or implemented by some combination.

[0157] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and the overall design constraints. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0158] Bus 1100 can be an Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), Extended Industry Standard Architecture (EISA), or another bus or bus-type structure. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one connection line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0159] Any process or method described in flowcharts or otherwise described herein can be understood as representing a module, segment, or portion of code that includes one or more executable instructions for implementing specific logical functions or steps, and the scope of preferred embodiments of the present disclosure includes additional implementations that can not be in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via the memory and / or communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps of the above-described methods can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above-described methods by any other appropriate means (e.g., by means of firmware).

[0160] The logic and / or steps represented in the flowcharts or otherwise described herein can be embodied in any readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with such an instruction execution system, apparatus, or device.

[0161] For the purposes of this specification, a "read only memory" can be any apparatus that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of the read only memory include the following: an electrical connection having one or more wires (electrical apparatus), a portable computer diskette (magnetic apparatus), a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or Flash memory), an optical fiber device, and a portable compact disc read only memory (CD ROM). In addition, a read only memory can even be paper or another suitable medium upon which the program can be printed, since the program can be electronically obtained, for example by optical scanning of the paper or other medium, followed by electronic means to

[0162] It should be understood that each part of the present disclosure can be realized by hardware, software, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be realized by software stored in a memory and executed by a suitable instruction execution system. For example, if realized by hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, application specific integrated circuit having a suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0163] Those skilled in the art of the present technology can understand that all or part of the steps of the above-described embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a readable storage medium, and when executed, include one or a combination of the steps of the method embodiments.

[0164] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software functional module. The integrated module, if realized in the form of a software functional module and sold or used as an independent product, can also be stored in a readable storage medium. The storage medium can be a read only memory, a magnetic disk or an optical disk, etc.

[0165] Figure 25 is a structure schematic diagram of a small target object detection device according to an embodiment of the present disclosure. As shown in FIG. 1, the small target object detection device includes a camera 10, a processor 20, a storage 30, and a display 40. Figure 25As shown, according to the small target object detection device 1000 of the present disclosure, the small target object detection device 1000 can include an acquisition module 1002, a first processing module 1004, a second processing module 1006, a third processing module 1008, and a determination module 1010.

[0166] The acquisition module 1002 is configured to acquire a detection image. The first processing module 1004 is configured to perform down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain a multi-scale shallow feature image and a deep feature image. The second processing module 1006 is configured to perform stripe feature extraction on the shallow feature image by using the small target detection model to obtain a stripe feature image. The third processing module 1008 is configured to perform semantic feature extraction on the deep feature image by using the small target detection model to obtain a semantic feature image. The determination module 1010 is configured to determine a small target object in the detection image according to the stripe feature image and the semantic feature image.

[0167] The present disclosure further provides an electronic device, which includes a memory storing execution instructions, and a processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module executes the small target object detection method described above.

[0168] The present disclosure further provides a readable storage medium, which stores execution instructions, and the execution instructions are executed by a processor to implement the small target object detection method described above.

[0169] In the description of the present specification, the description of the terms "one embodiment / way", "some embodiments / ways", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment / way or example are included in at least one embodiment / way or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily the same embodiment / way or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments / ways or examples. In addition, the person skilled in the art can combine and combine the different embodiments / ways or examples described in the present specification and the features of the different embodiments / ways or examples, without contradiction.

[0170] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0171] Those skilled in the art will understand that the above-described embodiments are merely intended to clarify the present disclosure, and are not intended to limit the scope of the present disclosure. Other changes or modifications can be made by those skilled in the art based on the above disclosure, and the changes or modifications are still within the scope of the present disclosure.

Claims

1. A small target object detection method characterized by, The method comprises the following steps: acquiring a detection image; performing down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain multi-scale shallow feature images and deep feature images; performing stripe feature extraction on the shallow feature images by using the small target detection model to obtain stripe feature images; performing semantic feature extraction on the deep feature images by using the small target detection model to obtain semantic feature images; determining small target objects in the detection image according to the stripe feature images and the semantic feature images; before the step of performing down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain multi-scale shallow feature images and deep feature images, the method further comprises the following steps: constructing a first neural network model, a second neural network model and a third neural network model, the first neural network model comprising a backbone network, a neck network and a small target object detection layer, the second neural network model comprising the backbone network, the neck network and a negative sample detection layer, the third neural network model comprising the backbone network, the neck network, the small target object detection layer and the negative sample detection layer; training the first neural network model by using a small target data set to obtain a trained first neural network model; training the second neural network model by using a negative sample data set to obtain a trained second neural network model; taking the backbone network weight, the neck network weight and the small target object detection layer weight of the trained first neural network model as the initial weight of the backbone network, the initial weight of the neck network and the initial weight of the small target object detection layer of the third neural network model; taking the negative sample detection layer weight of the trained second neural network model as the initial weight of the negative sample detection layer of the third neural network model; training the third neural network model by using the small target data set to obtain a trained third neural network model; taking the trained third neural network model as the small target detection model.

2. The small target object detection method according to claim 1, characterized by, The small target detection model comprises the backbone network, and the backbone network comprises a first convolution component and a second convolution component, and the step of performing down-sampling processing on the detection image in a backbone network by using a pre-trained small target detection model to obtain multi-scale shallow feature images and deep feature images comprises the following steps: calling the first convolution component to perform down-sampling processing on the detection image to obtain the shallow feature images; calling the second convolution component to perform down-sampling processing on the shallow feature images to obtain the deep feature images.

3. The small target object detection method according to claim 2, characterized in that, The second convolution component comprises a first convolution module, a first splicing module and a plurality of second convolution modules with different parameters, and the step of calling the second convolution component to perform down-sampling processing on the shallow feature images to obtain the deep feature images comprises the following steps: calling a plurality of second convolution modules to perform convolution processing on the shallow feature images respectively; calling the first splicing module to perform splicing processing on the shallow feature images after convolution to obtain a first splicing feature image; The first convolution module is called to perform convolution processing on the first spliced feature image to obtain the deep layer feature image.

4. The small target object detection method according to claim 1, characterized by, The small target detection model includes a stripe feature extraction component, the stripe feature extraction component includes a first max pooling module, a second max pooling module, a third convolution module and a second splicing module, the small target detection model is used to extract stripe features from the shallow layer feature image to obtain a stripe feature image, which includes: The first max pooling module and the second max pooling module are called to perform max pooling processing on the shallow layer feature image to obtain a first feature image and a second feature image, respectively; The third convolution module is called to perform convolution processing on the second feature image; The second splicing module is called to splice the first feature image and the second feature image after convolution processing to obtain the stripe feature image.

5. The small target object detection method according to claim 1, characterized in that, The small target detection model includes a semantic feature extraction component, the semantic feature extraction component includes a spatial attention module, the small target detection model is used to extract semantic features from the deep layer feature image to obtain a semantic feature image, which includes: The spatial attention module is called to locate the semantic features of the small target object in the deep layer feature image to obtain the semantic feature image.

6. The small target object detection method according to claim 1, characterized by, The small target detection model includes an up-sampling component, a first multi-feature fusion component and a first detection layer, and the small target object in the detection image is determined according to the stripe feature image and the semantic feature image, which includes: The up-sampling component is called to perform up-sampling combination processing on the deep layer feature image, so that the deep layer feature image after up-sampling combination processing has the same size as the stripe feature image; The first multi-feature fusion component is called to perform multi-feature fusion processing on the stripe feature image and the deep layer feature image after up-sampling combination processing to obtain a first multi-feature fusion image; The first detection layer is called to detect the small target object in the first multi-feature fusion image to determine the small target object in the detection image.

7. The small target object detection method according to claim 6, characterized in that, The up-sampling component includes a transpose convolution module, an up-sampling module and a third splicing module, and the up-sampling component is called to perform up-sampling combination processing on the deep layer feature image, which includes: The transpose convolution module is called to perform inverse convolution processing on the deep layer feature image to obtain a third feature image; The up-sampling module is called to perform up-sampling processing on the deep layer feature image to obtain a fourth feature image; The third splicing module is called to splice the third feature image and the fourth feature image to obtain the deep layer feature image after up-sampling combination processing.

8. The small target object detection method according to claim 1, characterized by, The small target detection model includes a third convolution component, a second multi-feature fusion component and a second detection layer, and the small target object in the detection image is determined according to the stripe feature image and the semantic feature image, which includes: The third convolution component is called to perform down-sampling processing on the stripe feature image, so that the stripe feature image after down-sampling processing has the same size as the semantic feature image; calling the second multi-feature fusion component to perform multi-feature fusion processing on the semantic feature image and the down-sampled stripe feature image to obtain a second multi-feature fusion image; calling the second detection layer to perform small target object detection on the second multi-feature fusion image to determine the small target object in the detection image.

9. The small target object detection method according to claim 1, wherein The training of the third neural network model using the small target dataset includes: freezing the weight of the negative sample detection layer of the third neural network model, and training the third neural network model using the small target dataset to obtain a to-be-corrected third neural network model; deleting the negative sample detection layer of the to-be-corrected third neural network model, and taking the to-be-corrected third neural network model after deleting the negative sample detection layer as the trained third neural network model.

10. The small target object detection method according to claim 1, characterized in that, After the training of the first neural network model using the small target dataset to obtain a trained first neural network model, the method further includes: performing small target object detection on the training images in the small target dataset using the trained first neural network model to obtain a small target object detection result; generating the negative sample dataset according to the small target object detection result, the negative sample being an object that is incorrectly identified as a small target object in the small target object detection result.

11. A small target object detection apparatus characterized by comprising: comprise: an acquisition module configured to acquire a detection image; a first processing module configured to perform down-sampling processing on the detection image in a backbone network using a pre-trained small target detection model to obtain multi-scale shallow feature images and deep feature images; a second processing module configured to perform stripe feature extraction on the shallow feature images using the small target detection model to obtain stripe feature images; a third processing module configured to perform semantic feature extraction on the deep feature images using the small target detection model to obtain semantic feature images; a determination module configured to determine a small target object in the detection image according to the stripe feature images and the semantic feature images; Before the pre-trained small target detection model is used to down-sample the detection image in the backbone network to obtain multi-scale shallow feature images and deep feature images, the method further comprises: constructing a first neural network model, a second neural network model and a third neural network model, wherein the first neural network model comprises a backbone network, a neck network and a small target object detection layer, the second neural network model comprises the backbone network, the neck network and a negative sample detection layer, and the third neural network model comprises the backbone network, the neck network, the small target object detection layer and the negative sample detection layer; training the first neural network model using a small target data set to obtain a trained first neural network model; training the second neural network model using a negative sample data set to obtain a trained second neural network model; using the backbone network weight, the neck network weight and the small target object detection layer weight of the trained first neural network model as the initial weight of the backbone network, the initial weight of the neck network and the initial weight of the small target object detection layer of the third neural network model; using the negative sample detection layer weight of the trained second neural network model as the initial weight of the negative sample detection layer of the third neural network model; training the third neural network model using the small target data set to obtain a trained third neural network model; and using the trained third neural network model as the small target detection model.

12. An electronic device, comprising: comprise: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the small target object detection method in any one of claims 1 to 10.

13. A readable storage medium, characterized by, The readable storage medium stores execution instructions, and the execution instructions are executed by the processor to implement the small target object detection method in any one of claims 1 to 10.

Images