Neural network-based target detection method, device and terminal equipment

Abstract

The application is suitable for the field of image processing, and provides a target detection method and device based on a neural network and a terminal device. The neural network comprises a plurality of feature extraction modules connected in sequence. Each feature extraction module comprises a target branch for performing image feature compression and excitation processing operations. The method comprises: inputting a to-be-detected image into the neural network; obtaining intermediate image processing results output by M feature extraction modules connected in sequence; obtaining a first detection result of image content with an area less than a threshold based on a fusion image obtained by fusing the M intermediate image processing results; obtaining a target image processing result output by N feature extraction modules after the connection order of the M feature extraction modules; and obtaining a second detection result of image content with an area greater than or equal to the threshold based on the target image processing result. The application can simultaneously detect large targets and small targets.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a target detection method, apparatus and terminal equipment based on neural networks. Background Technology

[0002] To ensure target detection accuracy, existing target detection networks are typically heavyweight networks running on large servers. Conversely, to improve the applicability of target detection networks, some also run on low-computing-power devices such as edge computing devices or embedded devices. At the same time, the number of parameters in these target detection networks cannot be too large. Therefore, target detection networks can only choose lightweight networks to run on low-computing-power devices. Currently, target detection networks running on low-computing-power devices tend to focus more on the detection of large targets, making it difficult to detect both large and small targets simultaneously. Furthermore, the detection accuracy when detecting large targets needs to be improved. Summary of the Invention

[0003] This application provides a target detection method, apparatus, and terminal device based on neural networks, which can solve the problem that existing target detection models have difficulty detecting both large and small targets simultaneously.

[0004] In a first aspect, embodiments of this application provide a target detection method based on a neural network, wherein the neural network includes a plurality of feature extraction modules connected in sequence, each feature extraction module includes a target branch, and the target branch performs image feature compression and excitation processing operations, the method comprising:

[0005] The image to be detected is input into the neural network, and the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network are obtained, and the target image processing result is obtained after the N feature extraction modules following the connection order of the M feature extraction modules in the neural network perform image processing in sequence; M and N are both positive integers.

[0006] The intermediate image processing results output by the M feature extraction modules are fused to obtain a fused image. Based on the fused image, the image content with an area smaller than a threshold in the image to be detected is detected to obtain a first detection result.

[0007] Based on the target image processing results, image content with an area greater than or equal to the threshold in the image to be detected is detected to obtain a second detection result.

[0008] Secondly, embodiments of this application provide a target detection device based on a neural network. The neural network includes multiple feature extraction modules connected in sequence. Each feature extraction module includes a target branch, and the target branch performs image feature compression and excitation processing operations. The device includes:

[0009] The result acquisition module is used to input the image to be detected into the neural network, acquire the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network, and acquire the target image processing result obtained by sequentially processing the image by the N feature extraction modules after the M feature extraction modules in the neural network; M and N are both positive integers.

[0010] The first detection module is used to fuse the intermediate image processing results output by the M feature extraction modules respectively to obtain a fused image, and to detect the image content with an area smaller than a threshold in the image to be detected based on the fused image to obtain a first detection result.

[0011] The second detection module is used to detect image content in the image to be detected whose area is greater than or equal to the threshold based on the target image processing result, and obtain a second detection result.

[0012] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the neural network-based target detection method described in the first aspect above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the neural network-based target detection method described in the first aspect above.

[0014] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the neural network-based target detection method described in the first aspect above.

[0015] The beneficial effects of this application embodiment compared with the prior art are as follows: This application detects image content (small targets) with an area smaller than a threshold in the image to be detected based on the intermediate image processing results output by M feature extraction modules connected sequentially in the neural network, obtaining a first detection result; wherein, each feature extraction module includes a target branch, and the target branch performs image feature compression and excitation processing operations; based on the target image processing results obtained by the N feature extraction modules after the connection order of the M feature extraction modules in the neural network sequentially performing image processing, image content (large targets) with an area larger than a threshold in the image to be detected is detected, obtaining a second detection result. The neural network in this application can obtain both small target detection results and large target detection results simultaneously after one inference, based on the intermediate image processing results of feature extraction modules located at different levels, resulting in high detection efficiency and saving computing resources. In addition, this application performs feature extraction based on feature extraction modules containing target branches, which can extract features related to large and small targets based on compression and excitation processing, reducing the influence of irrelevant features on the detection results and ensuring the detection accuracy of large and small targets. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the target detection method based on neural networks provided in this application;

[0018] Figure 2 This is a schematic diagram of a neural network structure provided in this application;

[0019] Figure 3 This is a schematic diagram of the feature extraction module provided in this application;

[0020] Figure 4 This is a schematic diagram of the processing flow of the SE unit provided in this application;

[0021] Figure 5 This is a visual illustration of the features extracted by the neural network at different downsampling factors provided in this application;

[0022] Figure 6 This is a schematic diagram of the process for obtaining the fused image provided in this application;

[0023] Figure 7This is a flowchart illustrating another neural network-based target detection method provided in this application;

[0024] Figure 8 This is a schematic diagram of another neural network structure provided in this application;

[0025] Figure 9 This is a schematic diagram of the structure of the first auxiliary feature extraction module provided in this application;

[0026] Figure 10 This is a schematic diagram of the complete structure of the neural network provided in this application;

[0027] Figure 11 This is a schematic diagram of the enhanced image obtained by the mosaic data enhancement method provided in this application;

[0028] Figure 12 This is a schematic diagram illustrating the convergence of the loss function during the neural network training process provided in this application.

[0029] Figure 13 This is a schematic structural block diagram of the target detection device based on a neural network provided in this application;

[0030] Figure 14 This is a schematic diagram of the terminal device provided in this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0033] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0034] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0035] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0037] Example 1:

[0038] Please see Figure 1 , Figure 1 The schematic flowchart of the target detection method based on a neural network provided in this application is shown. The neural network provided in this application includes multiple feature extraction modules connected in sequence. Each feature extraction module includes a target branch, which performs image feature compression and excitation processing operations.

[0039] Step 101: Input the image to be detected into the neural network, obtain the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network, and obtain the target image processing results obtained by the N feature extraction modules after the M feature extraction modules are connected in sequence in the neural network; M and N are both positive integers.

[0040] Before the feature extraction module, at least one other processing module can be connected in series in the neural network. That is, after the image to be detected is input into the neural network, it can be processed sequentially through other processing modules and the feature extraction module. In an optional implementation, the other processing module can be a convolutional processing module. Optionally, the neural network may contain two convolutional processing modules. See also... Figure 2 , Figure 2 This application provides a schematic diagram of a neural network structure. Figure 2 In this context, Input represents the image to be detected, conv1 and conv2 represent convolutional processing modules, and Res-SENet represents the feature extraction module. In practical applications, there can be one or more Res-SENet modules; this is just an example and not a limitation. Figure 2 The neural network shown contains three Res-SENets.

[0041] Optionally, the feature extraction module also includes a convolution branch connected in parallel with the target branch. The convolution branch is used to perform image convolution operations. The target branch includes an SE (Squeeze-and-Excitation) unit, a pooling unit, and a first convolution unit connected in sequence. The convolution branch includes at least one second convolution unit. The intermediate image processing result output by the feature extraction module is obtained by feature fusion of the image processing results of the target branch and the convolution branch.

[0042] In the target branch, the input features of the feature extraction module are first compressed and activated by the SE unit, then globally averaged by the pooling unit to reduce resolution, and finally convolved by the first convolutional unit to reduce the computational cost of the neural network. For the convolutional branch, the input features of the feature extraction module are convolved by at least one second convolutional unit, allowing for the extraction of as many features as possible to detect all image content in the image to be detected, thus improving the detection rate. It should be noted that the feature scale of the output image processing results of the target branch and the convolutional branch is the same; that is, the size and number of channels of the feature maps output by the two branches are the same. The intermediate image processing result output by the feature extraction module is obtained by feature fusion of the output image processing results of the target branch and the convolutional branch.

[0043] Feature extraction is performed based on the feature extraction module containing the target branch and the convolution branch in this application. On the one hand, extracting enough feature information based on the convolution branch can avoid missing the detection of image content. On the other hand, extracting features related to the image content based on the target branch can improve the detection accuracy while speeding up the detection speed of the neural network.

[0044] Please refer to this as an example rather than a limitation. Figure 3 , Figure 3 The diagram shows the structure of the feature extraction module provided in this application. The feature extraction module (Res-SENet) in this application is obtained by improving the basic network structure of ResNet. Figure 3 In this example, Input represents the input features of the feature extraction module, and Output represents the output features, i.e., the intermediate image processing result output by the feature extraction module. In this example, the pooling unit in the target branch uses a 2x2 kernel with a stride of 2, and the first convolutional unit uses a 1x1 kernel with a stride of 1. The convolutional branch contains three sequentially connected second convolutional units. Following the inference order of the convolutional branch, the kernel sizes of the three second convolutional units are 1x1, 3x3, and 1x1, respectively, with corresponding strides of 1, 2, and 1. It should be noted that... Figure 3 The numbers 104*104*64, 104*104*32, and 52*52*64 shown represent the feature scale.

[0045] Please see Figure 4 , Figure 4 The diagram illustrates the processing flow of the SE unit provided in this application. Specifically, the SE unit performs image feature compression and activation operations as follows: Global Maxpooling is applied to the input features to obtain a compressed feature map; a fully connected layer (FC) learns the correlation between channels in the compressed feature map to obtain an evaluation score for each channel; the evaluation score is then processed by a Sigmoid activation function to output the weight of each channel in the compressed feature map; this weight is multiplied by the input features and then output. The weight represents the importance of the channel; the larger the weight, the more relevant the feature of that channel is to the image content. Therefore, the SE unit can learn the correlation between channels, enabling subsequent networks to "pay attention" to the more important parts of the feature map (features relevant to the image content).

[0046] FLOPs (floating point operations) can be used to measure the complexity of a model, and their calculation formula is as follows:

[0047]

[0048] Where Depth represents the total number of all convolutional layers. This represents the square of the size of the output feature map of the i-th convolutional layer. C represents the square of the kernel size used in the i-th convolutional layer. i-1 and C iThese represent the number of channels in the input features and the number of channels in the output features of the i-th convolutional layer, respectively.

[0049] by Figure 3 Taking a feature scale of 104*104*64 for the input as an example, the FLOPs of the CSPBlock model are 7.421×108, while the FLOPs of the Res-SENet proposed in this embodiment are 7.132×107. The ratio of the FLOPs of Res-SENet to CSPBlock is approximately 1:10, thus the Res-SENet proposed in this application can significantly reduce the computational complexity of the neural network model.

[0050] Optionally, the feature extraction module performs downsampling processing on the image to be detected through the target branch. Before obtaining the intermediate image processing results output by the M feature extraction modules sequentially connected in the neural network, it further includes: selecting M feature extraction modules from the feature extraction modules included in the neural network with a downsampling factor of a set value. As an example and not a limitation, the set value can be 4, 8, or 16.

[0051] The reason why this application selects M feature extraction modules with a downsampling factor of a set value from the feature extraction modules contained in the neural network is as follows:

[0052] The purpose of obtaining the intermediate image processing results output by the M feature extraction modules in this application is to detect image content with an area smaller than the threshold in the image to be detected, that is, to detect small targets in the image to be detected.

[0053] In the neural network inference sequence, the downsampling factor for each feature extraction module gradually increases, and the resolution of the intermediate image processing result gradually decreases. The larger the downsampling factor, the lower the resolution, and the less obvious the features of small targets in the image to be detected. (See also...) Figure 5 , Figure 5 This paper presents a visualization diagram of the features extracted by the neural network under different downsampling factors provided in this application. In the diagram, L represents the inference depth in the inference order of the neural network corresponding to the output feature map of the Lth layer. The value of L ranges from [1, 5]. The larger the value of L, the deeper the inference depth and the larger the downsampling factor. The features of small targets in the extracted features are less obvious. Therefore, this application needs to obtain the shallow features extracted by the neural network to detect small targets in the image to be detected, that is, to obtain the intermediate image processing results output by the M feature extraction modules with smaller downsampling factors.

[0054] For image content with an area greater than or equal to the threshold in the image to be detected, that is, for large targets in the image to be detected, when the resolution of the intermediate image processing results is the same, the features of large targets in the intermediate image processing results are more obvious than the features of small targets. Therefore, when performing large target detection, we can obtain the deep features extracted by the neural network, that is, obtain the target image processing results obtained after the N feature extraction modules after the connection order of the M feature extraction modules in the neural network are processed in sequence.

[0055] Step 102: The intermediate image processing results output by the M feature extraction modules are fused to obtain a fused image. Based on the fused image, the image content with an area smaller than the threshold in the image to be detected is detected to obtain the first detection result.

[0056] The method described above for obtaining the intermediate image processing results output by the M feature extraction modules and obtaining the fused image can be called the shallow feature fusion method.

[0057] Optionally, before feature fusion, it is necessary to first unify the feature scales of the intermediate image processing results output by the M feature extraction modules. As one implementation method, the stride size corresponding to each of the M feature extraction modules can be calculated based on the intermediate image processing results output by each module and the unified feature scale; then, a 1*1 convolution kernel is used to perform convolution based on the stride size of each feature extraction module to achieve feature scale unification.

[0058] Optionally, if the downsampling factor corresponding to the selected M feature extraction modules fails to cover the set value, that is, if no feature extraction module with a downsampling factor of the set value is selected from the feature extraction modules included in the neural network, then H other processing modules with a downsampling factor of the set value can be selected from the other processing modules mentioned above, and feature fusion can be performed based on the image processing results output by the selected M feature extraction modules and H other processing modules to obtain a fused image; where H is a positive integer.

[0059] For example, if the values ​​are set to 4, 8, and 16, and only two feature extraction modules with downsampling factors of 8 and 16 are selected from the feature extraction modules included in the neural network, then another processing module with a downsampling factor of 4 can be selected from the other processing modules mentioned above. Assume... Figure 2 The downsampling factors for conv1 and conv2 are 2 and 4, respectively, while the downsampling factors for the three Res-SENets are 8, 16, and 32, respectively. Therefore, conv2, along with the first two Res-SENets in the neural network inference order, are ultimately selected. For details on obtaining the fused image based on the outputs of conv2 and the first two Res-SENets in the neural network inference order, please refer to [link to documentation / reference]. Figure 6 , Figure 6 A schematic diagram of the process for acquiring the fused image is shown.

[0060] Optionally, the first detection result may indicate the location of image content in the image to be detected with an area smaller than a threshold. By way of example and not limitation, the first detection result may include at least one detection box containing image content with an area smaller than the threshold, and may also include at least one keypoint located at the center of the image content with an area smaller than the threshold.

[0061] It should be noted that the feature fusion mentioned in this application can refer to either the concat operation or the add operation.

[0062] Step 103: Based on the target image processing results, detect the image content in the image to be detected whose area is greater than or equal to the threshold, and obtain the second detection result.

[0063] Optionally, after obtaining the target image processing result, the target image processing result can be convolved to obtain a convolutional image processing result, and a second detection result can be obtained based on the convolutional image processing result.

[0064] Compared to existing methods that use two separate neural networks for small and large object detection, the neural network described in this application can simultaneously obtain small and large object detection results after a single inference step, resulting in high detection efficiency and saving computational resources.

[0065] As an example and not a limitation, vehicle detection can be performed using the neural network provided in this application. Vehicles with an area smaller than a threshold in the image to be detected are considered small targets, and vehicles with an area greater than or equal to the threshold are considered large targets.

[0066] This application detects image content (small targets) with an area smaller than a threshold in the image to be detected based on the intermediate image processing results output by M feature extraction modules connected sequentially in a neural network, obtaining a first detection result. Each feature extraction module includes a target branch, which performs image feature compression and excitation processing. Based on the target image processing results obtained from the sequential image processing of N feature extraction modules following the M feature extraction modules in the neural network, image content (large targets) with an area larger than the threshold in the image to be detected is detected, obtaining a second detection result. The neural network in this application can simultaneously obtain small target detection results and large target detection results after one inference operation, resulting in high detection efficiency and saving computational resources. Furthermore, this application performs feature extraction based on feature extraction modules containing target branches, which can extract features related to both large and small targets based on compression and excitation processing, reducing the influence of irrelevant features on the detection results and ensuring the detection accuracy of both large and small targets.

[0067] Example 2:

[0068] Please see Figure 7 , Figure 7 This illustration shows a schematic flow of the neural network-based target detection method provided in this application. The neural network provided in this application includes N feature extraction modules, with a first auxiliary feature extraction module connected in parallel at the input end and a second auxiliary feature extraction module connected in parallel at the output end. The first and second auxiliary feature extraction modules are used to extract global features.

[0069] Please see Figure 8 , Figure 8 A schematic diagram of another neural network structure provided in this application is shown. Res-SENet can reduce network inference time, but the detection accuracy of neural networks containing Res-SENet still has the potential for improvement. Therefore, a first auxiliary feature extraction module can be connected in parallel at the input of N feature extraction modules, and a second auxiliary feature extraction module can be connected in parallel at the output to mine deeper global features. In object detection, global information helps improve the accuracy of object detection; therefore, this application adds some auxiliary feature extraction modules to the backbone of the neural network, enabling the neural network to learn more feature information.

[0070] In one optional embodiment, both the first auxiliary feature extraction module and the second auxiliary feature extraction module include an auxiliary convolution unit, a channel attention unit, and a spatial attention unit connected in sequence; wherein, the kernel size of the auxiliary convolution unit included in the first auxiliary feature extraction module is larger than the kernel size of the auxiliary convolution unit included in the second auxiliary feature extraction module.

[0071] The first and second auxiliary feature extraction modules process the image sequentially through auxiliary convolution units, channel attention units, and spatial attention units to obtain the global features of the image as output.

[0072] As the inference depth of the neural network increases, the downsampling factor increases, and the resolution of the resulting image processing (feature map) decreases. The receptive field of the same convolutional kernel on a lower-resolution feature map increases. The resolution of the output features of the N feature extraction modules is lower than the resolution of the input features. To ensure feature fusion, the resolution of the output features of the second auxiliary feature extraction module must be lower than the resolution of the output features of the first auxiliary feature extraction module. However, the first and second auxiliary feature extraction modules do not change the feature scale of the input features (i.e., the feature scale of the input and output features of the auxiliary feature extraction modules is consistent). Therefore, the resolution of the input features of the second auxiliary feature extraction module is lower than that of the first auxiliary feature extraction module. Consequently, to ensure that the convolutional kernels in the first and second auxiliary feature extraction modules have similar receptive field sizes, the kernel size of the auxiliary convolutional units in the first auxiliary feature extraction module must be larger than the kernel size of the auxiliary convolutional units in the second auxiliary feature extraction module.

[0073] In another optional embodiment, both the first auxiliary feature extraction module and the second auxiliary feature extraction module include a target auxiliary convolutional unit, a channel attention unit, and a spatial attention unit connected in sequence; the target auxiliary convolutional unit includes a first auxiliary convolutional unit and a second auxiliary convolutional unit connected in parallel; wherein, the kernel size of the first auxiliary convolutional unit included in the first auxiliary feature extraction module is larger than the kernel size of the first auxiliary convolutional unit included in the second auxiliary feature extraction module, and the kernel size of the second auxiliary convolutional unit included in the first auxiliary feature extraction module is larger than the kernel size of the second auxiliary convolutional unit included in the second auxiliary feature extraction module.

[0074] The first auxiliary feature extraction module and the second auxiliary feature extraction module convolve the input features through the first auxiliary convolution unit and the second auxiliary convolution unit to obtain the first auxiliary convolution feature and the second auxiliary convolution feature. Then, the first auxiliary convolution feature and the second auxiliary convolution feature after feature fusion are processed sequentially through the channel attention unit and the spatial attention unit to obtain the global features of the image as the output.

[0075] In order for the first auxiliary feature extraction module and the second auxiliary feature extraction module to extract more global information, the kernel sizes of the first auxiliary convolution unit and the second auxiliary convolution unit in the first auxiliary feature extraction module should be different.

[0076] Optionally, the kernel size of the first auxiliary convolutional unit in the first auxiliary feature extraction module can be 3*3, and the kernel size of the second auxiliary convolutional unit can be 5*5; the kernel size of the first auxiliary convolutional unit in the second auxiliary feature extraction module can be 1*1, and the kernel size of the second auxiliary convolutional unit can be 3*3. The downsampling factor of the first auxiliary feature extraction module can be 16, and the downsampling factor of the second auxiliary feature extraction module can be 32.

[0077] Specifically, the implementation process of the neural network-based target detection method provided in this embodiment is as follows:

[0078] Step 701: Input the image to be detected into the neural network and obtain the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network.

[0079] For details regarding this step, please refer to the relevant description in step 101.

[0080] Step 702: Based on the first auxiliary feature extraction module, perform feature extraction on the target intermediate image processing result output by the Mth feature extraction module to obtain the first extraction result; perform feature fusion between the first extraction result and the target intermediate image processing result to obtain the first fused image processing result.

[0081] After obtaining the first fused image processing result, the first fused image processing result is output to the input terminals of the above N feature extraction modules.

[0082] Please see Figure 9 , Figure 9 The diagram illustrates the structure of the first auxiliary feature extraction module provided in this application. The second auxiliary feature extraction module can be obtained by modifying the sizes of the two convolutional kernels in the diagram accordingly; this embodiment will not illustrate it further. In this embodiment, the input features are processed by two convolutional kernels of different sizes and concatenated to form new features. Then, using the CBAM concept, an attention mechanism is introduced into the auxiliary module to extract more effective information. The attention mechanism allows the network to focus more on effective features while suppressing ineffective features. The spatial attention model focuses on the "location" of the target in the image, while the channel attention model focuses on the "content" of the target that may exist in the image.

[0083] For the auxiliary feature extraction module in this embodiment, the outputs of its channel attention unit and spatial attention unit are represented as follows:

[0084]

[0085]

[0086] Where F is the input feature map of the auxiliary feature extraction module, and M... c Here, F' is the channel attention map, and M is the output feature map of the channel attention unit. s This is a 2D spatial attention map, where F″ is the output feature map of the spatial attention unit. This indicates element-wise multiplication.

[0087] Step 703: Obtain the first output result after the N feature extraction modules sequentially process the first fused image processing result.

[0088] It should be noted that, based on the first output result, image content with an area greater than or equal to the threshold in the image to be detected can be detected to obtain the second detection result.

[0089] Step 704: Based on the first extraction result, determine the input data for the second auxiliary feature extraction module.

[0090] In an alternative implementation, the first extraction result can be determined as the input data of the second auxiliary feature extraction module.

[0091] It should be noted that, in order to ensure that the output features of the second auxiliary feature extraction module have the same feature scale as the first output result, the first extraction result needs to undergo average pooling processing before being input into the second auxiliary feature extraction module.

[0092] In another optional embodiment, a third auxiliary feature extraction module is connected in parallel on at least one connection line between the N feature extraction modules; the first auxiliary feature extraction module, the third auxiliary feature extraction module, and the second auxiliary feature extraction module are connected in series in sequence; based on the first extraction result, the input data of the second auxiliary feature extraction module is determined, including: performing feature extraction on the first extraction result based on the third auxiliary feature extraction module to obtain a third extraction result; and determining the third extraction result as the input data of the second auxiliary feature extraction module.

[0093] Step 705: Obtain the second extraction result obtained by the second auxiliary feature extraction module from the input data; fuse the second extraction result with the first output result to obtain the target image processing result.

[0094] Step 706: The intermediate image processing results output by the M feature extraction modules are fused to obtain a fused image. Based on the fused image, the image content with an area smaller than the threshold in the image to be detected is detected to obtain the first detection result.

[0095] After obtaining the fused image, a convolution operation can be performed on it. This convolution operation is a crucial part, used to extract key information from the fused image. In the design of this part, a choice is made between channel attention mechanism and ordinary convolution module. It is generally believed that the channel attention model can better focus on the features of the object. However, in the experiment, it was found that the accuracy of using ordinary convolution and channel attention model is almost the same, and the speed of ordinary convolution is faster than channel attention. Therefore, this application uses a 3*3 convolution kernel to convolve the fused image to obtain a fused feature map, and then obtains the first detection result based on the fused feature map.

[0096] Step 707: Based on the target image processing results, detect image content in the image to be detected whose area is greater than or equal to the threshold, and obtain the second detection result.

[0097] For details regarding steps 706 and 707, please refer to the relevant descriptions in steps 102 and 102.

[0098] Alternatively, based on the above description, one complete structural diagram of this application can be obtained, as shown in the figure below. Figure 10 , Figure 10 The diagram shows a complete structure of a neural network provided in this application. The neural network contains three feature extraction modules, where M is 2 and N is 1.

[0099] Optionally, after obtaining the second detection result, the method further includes:

[0100] Based on the first detection result, first image content with an area smaller than a threshold in the image to be detected is obtained, and third image content with an area smaller than a first area threshold is filtered from the first image content; wherein, the first area threshold can be the average area of ​​the first image content.

[0101] And / or, based on the second detection result, obtain second image content in the image to be detected whose area is less than a threshold, and filter third image content whose area is less than a second area threshold from the second image content; wherein, the second area threshold can be the average area of ​​the second image content.

[0102] The content of the third image is filled into the background region of the image to be detected to obtain the enhanced image; the background region is the region that does not contain the content of the first image and the content of the second image.

[0103] Based on the enhanced image, a target training sample set is obtained. The target training sample set is obtained by updating the initial training sample set containing the image to be detected based on the enhanced image. The target training sample set is used for iterative model training of the neural network, or for model training of other neural networks.

[0104] Optionally, in order to improve the detection accuracy of the neural network when detecting small targets, third image content with an area smaller than a first area threshold, selected from the first image content, can be used to fill the background area of ​​the image to be detected.

[0105] Background areas in the image to be detected are not conducive to the learning of the neural network during the training process. Therefore, this application adopts a "brute force" data augmentation method to increase the amount of image content in the background area of ​​an image by directly copying the third image content in the image to other background areas of the image.

[0106] Optionally, the third image content can be filled into the background area of ​​the image to be detected. Third image content with an area smaller than the background area can be filled into the background area in ascending or descending order of area. If, after all the third image content with an area smaller than the background area has been filled, there are still fillable areas in the background area, the third image content with an area smaller than the background area can be filled again in ascending or descending order of area until the area of ​​the fillable area in the background area is smaller than the minimum area of ​​the third image content.

[0107] Optionally, after obtaining the target training sample set, further mosaic data augmentation can be performed on the target training sample set: four image samples are randomly selected from the target training sample set, and the four image samples are placed in the upper left, lower left, upper right, and lower right areas of the preset mosaic canvas, respectively. Then, they are cropped and spliced ​​to obtain the newly augmented image; based on the newly augmented image, the target training sample set is updated.

[0108] Mosaic data augmentation involves randomly selecting four image samples from the target training sample set, performing conventional data augmentation (such as color transformation, cropping, inversion, etc.) and / or "brute force" data augmentation, scaling the images by a certain ratio with a scaling factor limited to [0.5, 0.7], placing the four images at the four corners of a preset mosaic canvas, cropping and stitching the images according to occlusion, and finally forming a new augmented image.

[0109] Please see Figure 11 , Figure 11The diagram illustrates the enhanced image obtained using the mosaic data augmentation method, i.e., the newly added enhanced image. It can be observed that compared to the four selected image samples (images 1 to 4), a few targets in the newly enhanced image disappear directly from the image due to cropping, while some targets remain in the stitched image. Furthermore, some target bounding boxes are cropped so that only a portion of the target exists in the stitched image; these targets require filtering. For all targets in the newly enhanced image, those with an IoU value less than 0.3 before and after augmentation can be directly labeled as background. In the newly enhanced image, compared to image 2, the three smaller bounding boxes in the lower left corner of the image are retained in the stitched image due to cropping, resulting in an area less than 0.3 of the original image. Therefore, these targets can be considered severely occluded and treated as background. The mosaic data augmentation method can directly enrich the background of the dataset and increase the number and distribution of small targets in the dataset. Meanwhile, data augmentation does not change the size of the model itself. Therefore, effective data augmentation methods can improve the accuracy of object detection without increasing the inference time of the model.

[0110] Please see Figure 12 , Figure 12 The diagram illustrates the convergence of the loss function during neural network training. Compared to the Tiny-YOLOv4 model, the neural network model in this embodiment converges faster, starting to converge around 150 iterations and stabilizing around 400 iterations. Tiny-YOLOv4, on the other hand, begins to converge around 400 iterations, and its loss curve is more stable after convergence. The neural network model in this embodiment exhibits slight oscillations after convergence, but the convergence loss is lower than that of Tiny-YOLOv4, with the final loss approaching 0.1. This indicates that the neural network model in this embodiment performs better in detecting small targets. Due to the effect of data augmentation, there are more small target data in the target training sample set during training. The detection of small targets is the biggest factor affecting the loss of the neural network model. Large targets are easier to detect, so their proportion in the loss is relatively small. After the neural network model tends to converge, large targets in the target training sample set can be detected by both detection models under the same hyperparameters. The same small targets can be detected better under the model of this application. Therefore, the final convergence loss is better than that of the Tiny-YOLOV4 model.

[0111] Under the same testing environment, PR curves for targets of different sizes were plotted using different models, i.e., curves with precision (Precise) on the horizontal axis and recall (Recall) on the vertical axis. For large targets, the two curves are roughly the same, with the model in this embodiment slightly outperforming Tiny-YOLOV4. The results show that the two models have roughly the same detection ability for large targets. For small targets, the model in this embodiment is significantly better than Tiny-YOLOV4. The intersection of the curve of the model in this embodiment with the horizontal axis is also closer to 1. In addition, the recall of the model in this embodiment is also better than that of the Tiny-YOLOV4 model. In summary, compared with Tiny-YOLOV4, the improved model in this embodiment has better detection accuracy for small targets, that is, the improved method in this embodiment can improve the detection accuracy of small targets.

[0112] To verify the impact of various improvement strategies based on Tiny-YOLOV4 on the actual performance of the model in this embodiment, multiple ablation experiments were conducted. The main structures tested in the ablation experiments included the Res-SENet designed in this embodiment, the auxiliary feature extraction module, the shallow feature fusion method, the hybrid mosaic and brute-force data augmentation method, and the ResNet-D module. The baseline model was Tiny-YOLOV4, with an mAP (Mean Average Precision) of 83.21%. The design and results of the ablation experiments are shown in Table 1. In the table, "√" indicates that the structure or method was used in the model, and "×" indicates that it was not used. In the first and second sets of experiments, the CSPBlock in Tiny-YOLOV4 was replaced with the Res-SENet and ResNet-D structures in this embodiment, respectively. Compared to the CSPBlock module, the Res-SENet structure achieved a 2.16% improvement, and ResNet-D achieved a 1.03% improvement. The experimental results show that the Res-SENet structure achieved better detection performance than ResNet-D and CSPBlock.

[0113] In the ablation experiments of groups 1 and 3, and groups 2 and 10, auxiliary feature extraction modules were added to the Res-SENet and ResNet-D structures, respectively. Compared with group 1, the mAP of group 3 increased by 0.34%, and the mAP of group 10 increased by 0.12% compared with group 2. The experimental results show that the auxiliary module based on the attention mechanism can bring a small improvement to the network, bringing a 0.34% accuracy improvement on the basis of the Res-SENet module. In experiments groups 1, 3, 4, 6, 2, and 9, shallowly extracted features were fused in different model structures, and predictions were made based on the fused features. Group 4 added a shallow feature fusion method to the experiment in group 1, group 6 added a shallow feature fusion method to the experiment in group 3, and group 9 added a shallow feature fusion method to the network structure of ResNet-D. Compared to group 1, the shallow feature fusion method in group 4 achieved a 0.98% improvement in mAP. In the experiment in group 6, the model's accuracy improved by 1.42% compared to group 3. This demonstrates that the shallow feature fusion method itself can improve object detection performance, and its combination with the auxiliary feature extraction module and Res-SENet can further improve the network performance. In the experiment in group 9, the shallow feature fusion method still brought a 1.09% improvement in accuracy to the model in the network structure based on ResNet-D.

[0114] Table 1. Multiple ablation experimental combinations and their results

[0115]

[0116] Shallow feature fusion methods can be easily inserted into various network structures. Therefore, in experiment group 11, the shallow features of the Tiny-YOLOV4 model were directly fused before prediction. The shallow feature fusion method still achieved a 0.77% improvement in the CSPBlock-based Tiny-YOLOV4 model, further demonstrating that shallow feature fusion can also improve detection accuracy, especially for small targets. The data augmentation in the Tiny-YOLOV4 model uses a mosaic data augmentation method. In experiments groups 1 and 5, and groups 6 and 7, a hybrid mosaic and brute-force data augmentation method was used during the data loading stage. Experimental results show that the hybrid mosaic and brute-force data augmentation method achieved a slight improvement in model accuracy. In summary, the ablation experiments further demonstrate that the Res-SENet structure, auxiliary feature extraction module, shallow feature fusion method, hybrid mosaic, and brute-force data augmentation method in this embodiment can all improve the model's accuracy. Among these, the Res-SENet structure and shallow feature fusion method have the greatest impact on the model's accuracy improvement, with the shallow feature fusion method's improvement mainly stemming from small targets. The auxiliary feature extraction module and data augmentation method also contribute to improving the algorithm's accuracy to some extent.

[0117] The neural network provided in this application includes N feature extraction modules, with a first auxiliary feature extraction module connected in parallel at the input end and a second auxiliary feature extraction module connected in parallel at the output end. This application inputs the image to be detected into the neural network. Based on the intermediate image processing results output by the M feature extraction modules sequentially connected in the neural network, a first detection result is obtained. Based on the first auxiliary feature extraction module, features are extracted from the intermediate image processing result output by the Mth feature extraction module to obtain a first extraction result. The first extraction result is fused with the intermediate image processing result to obtain a first fused image processing result. A first output result is obtained after the N feature extraction modules sequentially process the first fused image processing result. Based on the first extraction result, the input data for the second auxiliary feature extraction module is determined. A second extraction result is obtained by the second auxiliary feature extraction module extracting features from the input data. The second extraction result is fused with the first output result to obtain a target image processing result. Based on the target image processing result, a second detection result is obtained. This application adds an auxiliary feature extraction module to the neural network to extract more global information, thereby further improving the detection accuracy of the neural network when detecting large targets based on the global information.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0119] Example 3:

[0120] Please see Figure 13 , Figure 13 A schematic structure of the target detection device based on a neural network provided in this application is shown. For ease of explanation, only the parts relevant to the embodiments of this application are shown in the figure. The neural network includes multiple feature extraction modules connected in sequence, each feature extraction module including a target branch, which performs image feature compression and excitation processing operations.

[0121] Reference Figure 13 The device includes a result acquisition module 131, a first detection module 132, and a second detection module 133; the specific functions of each module are as follows:

[0122] The result acquisition module 131 is used to input the image to be detected into the neural network, acquire the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network, and acquire the target image processing results obtained by the N feature extraction modules after the M feature extraction modules in the neural network are connected in sequence and then processed in sequence; M and N are both positive integers.

[0123] The first detection module 132 is used to fuse the intermediate image processing results output by the M feature extraction modules to obtain a fused image, and to detect image content with an area smaller than a threshold in the image to be detected based on the fused image to obtain a first detection result.

[0124] The second detection module 133 is used to detect image content in the image to be detected with an area greater than or equal to a threshold based on the target image processing results, and obtain a second detection result.

[0125] Optionally, a first auxiliary feature extraction module is connected in parallel to the input of the N feature extraction modules, and a second auxiliary feature extraction module is connected in parallel to the output of the N feature extraction modules. The first and second auxiliary feature extraction modules are used to extract global features; the result acquisition module 131 is specifically used for:

[0126] Based on the first auxiliary feature extraction module, feature extraction is performed on the target intermediate image processing result output by the Mth feature extraction module to obtain the first extraction result;

[0127] The first extraction result is fused with the target intermediate image processing result to obtain the first fused image processing result;

[0128] Obtain the first output result after sequentially processing the first fused image by N feature extraction modules;

[0129] Based on the first extraction result, determine the input data for the second auxiliary feature extraction module;

[0130] Obtain the second extraction result obtained by the second auxiliary feature extraction module from the input data;

[0131] The second extraction result is fused with the first output result to obtain the target image processing result.

[0132] Optionally, both the first auxiliary feature extraction module and the second auxiliary feature extraction module include an auxiliary convolution unit, a channel attention unit, and a spatial attention unit connected in sequence; wherein, the kernel size of the auxiliary convolution unit included in the first auxiliary feature extraction module is larger than the kernel size of the auxiliary convolution unit included in the second auxiliary feature extraction module.

[0133] Optionally, both the first auxiliary feature extraction module and the second auxiliary feature extraction module include a target auxiliary convolutional unit, a channel attention unit, and a spatial attention unit connected in sequence; the target auxiliary convolutional unit includes a first auxiliary convolutional unit and a second auxiliary convolutional unit connected in parallel; wherein, the kernel size of the first auxiliary convolutional unit in the first auxiliary feature extraction module is larger than the kernel size of the first auxiliary convolutional unit in the second auxiliary feature extraction module, and the kernel size of the second auxiliary convolutional unit in the first auxiliary feature extraction module is larger than the kernel size of the second auxiliary convolutional unit in the second auxiliary feature extraction module.

[0134] The feature extraction module downsamples the image to be detected through the target branch. Optionally, the device further includes a selection module, which is used for:

[0135] Before obtaining the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network, select M feature extraction modules with a downsampling factor of a set value from the feature extraction modules contained in the neural network.

[0136] Optionally, the feature extraction module further includes a convolutional branch connected in parallel with the target branch. The convolutional branch is used to perform image convolution operations. The target branch includes an SE unit, a pooling unit, and a first convolutional unit connected in sequence. The convolutional branch includes at least one second convolutional unit. The intermediate image processing result output by the feature extraction module is obtained by feature fusion of the image processing results of the target branch and the convolutional branch.

[0137] Optionally, the apparatus further includes a sample set acquisition module, which is used for:

[0138] Based on the first detection result, obtain the first image content in the image to be detected whose area is less than the threshold, and filter the third image content whose area is less than the first area threshold from the first image content.

[0139] And / or, based on the second detection result, obtain the second image content in the image to be detected whose area is less than a threshold, and filter the third image content whose area is less than the second area threshold from the second image content;

[0140] The content of the third image is filled into the background region of the image to be detected to obtain the enhanced image; the background region is the region that does not contain the content of the first image and the content of the second image.

[0141] Based on the enhanced image, a target training sample set is obtained. The target training sample set is obtained by updating the initial training sample set containing the image to be detected based on the enhanced image. The target training sample set is used for iterative model training of the neural network, or for model training of other neural networks.

[0142] The target detection device based on neural networks provided in this application can be applied in the aforementioned method embodiments one and two. For details, please refer to the descriptions of the aforementioned method embodiments one and two, which will not be repeated here.

[0143] Example 4:

[0144] Please see Figure 14 , Figure 14 A schematic structure of a terminal device according to an embodiment of this application is shown. The terminal device 14 of this embodiment includes: at least one processor 140 ( Figure 14 The above describes the process of target detection based on neural networks in Embodiments 1 and 2, which is shown in only one example. The processor 140 executes the computer program 142 to implement the steps of the neural network-based target detection method in Embodiments 1 and 2.

[0145] The terminal device 14 may be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor 140 and a memory 141. Those skilled in the art will understand that... Figure 14 This is merely an example of terminal device 14 and does not constitute a limitation on terminal device 14. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0146] The processor 140 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0147] In some embodiments, the memory 141 may be an internal storage unit of the terminal device 14, such as a hard disk or memory of the terminal device 14. In other embodiments, the memory 141 may be an external storage device of the terminal device 14, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 14. Furthermore, the memory 141 may include both internal and external storage units of the terminal device 14. The memory 141 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 141 can also be used to temporarily store data that has been output or will be output.

[0148] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a terminal device, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target detection method based on neural networks, characterized in that, The neural network includes multiple feature extraction modules connected in sequence. Each feature extraction module includes a target branch, which performs image feature compression and excitation processing operations. The method includes: The image to be detected is input into the neural network, and the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network are obtained, and the target image processing result is obtained after the N feature extraction modules following the connection order of the M feature extraction modules in the neural network perform image processing in sequence; M and N are both positive integers. The intermediate image processing results output by the M feature extraction modules are fused to obtain a fused image. Based on the fused image, the image content with an area smaller than a threshold in the image to be detected is detected to obtain a first detection result. Based on the target image processing result, image content with an area greater than or equal to the threshold in the image to be detected is detected to obtain a second detection result; The N feature extraction modules are connected in parallel with a first auxiliary feature extraction module at their inputs and in parallel with a second auxiliary feature extraction module at their outputs. The first and second auxiliary feature extraction modules are used for global feature extraction. The step of obtaining the target image processing result obtained by sequentially processing the N feature extraction modules following the M feature extraction modules in the neural network includes: extracting features from the intermediate target image processing result output by the Mth feature extraction module based on the first auxiliary feature extraction module to obtain a first extraction result; fusing the first extraction result with the intermediate target image processing result to obtain a first fused image processing result; obtaining a first output result output by the N feature extraction modules sequentially processing the first fused image processing result; determining the input data of the second auxiliary feature extraction module based on the first extraction result; obtaining a second extraction result obtained by the second auxiliary feature extraction module extracting features from the input data; and fusing the second extraction result with the first output result to obtain the target image processing result.

2. The method as described in claim 1, characterized in that, Both the first auxiliary feature extraction module and the second auxiliary feature extraction module include an auxiliary convolution unit, a channel attention unit, and a spatial attention unit connected in sequence; wherein, the kernel size of the auxiliary convolution unit in the first auxiliary feature extraction module is larger than the kernel size of the auxiliary convolution unit in the second auxiliary feature extraction module.

3. The method as described in claim 1, characterized in that, Both the first auxiliary feature extraction module and the second auxiliary feature extraction module include a target auxiliary convolutional unit, a channel attention unit, and a spatial attention unit connected in sequence; the target auxiliary convolutional unit includes a first auxiliary convolutional unit and a second auxiliary convolutional unit connected in parallel; Wherein, the kernel size of the first auxiliary convolutional unit included in the first auxiliary feature extraction module is larger than the kernel size of the first auxiliary convolutional unit included in the second auxiliary feature extraction module, and the kernel size of the second auxiliary convolutional unit included in the first auxiliary feature extraction module is larger than the kernel size of the second auxiliary convolutional unit included in the second auxiliary feature extraction module.

4. The method as described in claim 1, characterized in that, The feature extraction module performs downsampling processing on the image to be detected through the target branch. Before obtaining the intermediate image processing results output by the M feature extraction modules sequentially connected in the neural network, the module further includes: Select M feature extraction modules from the feature extraction modules included in the neural network, each with a downsampling factor of a set value.

5. The method as described in claim 1, characterized in that, The feature extraction module also includes a convolution branch connected in parallel with the target branch. The convolution branch is used to perform image convolution operations. The target branch includes an SE unit, a pooling unit, and a first convolution unit connected in sequence. The convolution branch includes at least one second convolution unit. The intermediate image processing result output by the feature extraction module is obtained by feature fusion of the image processing results of the target branch and the convolution branch.

6. The method as described in claim 1, characterized in that, After obtaining the second detection result, the method further includes: Based on the first detection result, obtain the first image content in the image to be detected whose area is less than a threshold, and filter the third image content whose area is less than the first area threshold from the first image content. And / or, based on the second detection result, obtain the second image content in the image to be detected whose area is less than a threshold, and filter the third image content whose area is less than the second area threshold from the second image content; The third image content is filled into the background region of the image to be detected to obtain an enhanced image; the background region is a region that does not contain the first image content and the second image content. Based on the enhanced image, a target training sample set is obtained, which is obtained by updating the initial training sample set containing the image to be detected based on the enhanced image; the target training sample set is used for iterative model training of the neural network, or the target training sample set is used for model training of other neural networks.

7. A target detection device based on a neural network, characterized in that, The neural network includes multiple feature extraction modules connected in sequence. Each feature extraction module includes a target branch, which performs image feature compression and excitation processing operations. The device includes: The result acquisition module is used to input the image to be detected into the neural network, acquire the intermediate image processing results output by the M feature extraction modules connected in sequence in the neural network, and acquire the target image processing result obtained by sequentially processing the image by the N feature extraction modules after the M feature extraction modules in the neural network; M and N are both positive integers. The first detection module is used to fuse the intermediate image processing results output by the M feature extraction modules respectively to obtain a fused image, and to detect the image content with an area smaller than a threshold in the image to be detected based on the fused image to obtain a first detection result. The second detection module is used to detect image content in the image to be detected whose area is greater than or equal to the threshold based on the target image processing result, and to obtain a second detection result; The N feature extraction modules are connected in parallel with a first auxiliary feature extraction module at their input ends and a second auxiliary feature extraction module at their output ends. The first and second auxiliary feature extraction modules are used for global feature extraction. The result acquisition module is further specifically used for: extracting features from the target intermediate image processing result output by the Mth feature extraction module based on the first auxiliary feature extraction module to obtain a first extraction result; fusing the first extraction result with the target intermediate image processing result to obtain a first fused image processing result; acquiring a first output result output by the N feature extraction modules after sequentially processing the first fused image processing result; determining the input data of the second auxiliary feature extraction module based on the first extraction result; acquiring a second extraction result obtained by the second auxiliary feature extraction module extracting features from the input data; and fusing the second extraction result with the first output result to obtain the target image processing result.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images