Optimization processing method and device of YOLO model applied to target detection

Abstract

The invention relates to the technical field of image recognition, and particularly discloses a YOLO model optimization processing method and device applied to target detection, an attention module is integrated into a backbone network and / or a multi-scale fusion network of a YOLO model, the YOLO model is optimized, the YOLO model emphasizing important feature regions in an image can be obtained, and the target detection accuracy is improved. Therefore, when the YOLO model carries out target identification in the image, small targets and edge targets in the low-resolution image can be accurately identified, and the structural information of the targets in the high-resolution image can be accurately extracted, so that the identification accuracy and efficiency of the targets of various sizes in the image can be improved, and the demand of refined inspection tasks can be met; the optimized YOLO model has the advantages of high precision and light weight, can be flexibly deployed in embedded equipment and a server side, and meets the target identification requirements of various scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an optimization processing method and device for a YOLO model applied to target detection. Background Art

[0002] With the rapid development of unmanned aerial vehicle (UAV) technology, the application requirements of UAVs in fields such as border monitoring, power line inspection, and forest fire prevention are increasing continuously. However, in these actual scenarios, the computing resources and transmission capabilities of UAVs limit the efficiency and accuracy of target detection and recognition systems.

[0003] Currently, UAVs usually carry embedded devices and optoelectronic pods to perform real-time target detection tasks. However, the computing capabilities of these embedded devices are limited, and it is difficult to run target detection models with high computational complexity, such as the DETR model obtained based on the Transformer detection algorithm. And to ensure real-time performance, lightweight models, such as the YOLO model, are usually selected for target detection of video data on the airborne side.

[0004] However, it is found in practice that existing YOLO models usually cannot identify small targets or edge targets in low-resolution video data and cannot accurately extract the structural information of targets in high-resolution images, making it difficult to meet the requirements of refined inspection tasks. Therefore, there is an urgent need to propose a technical solution that can accurately identify small targets and edge targets in low-resolution images and accurately extract the structural information of targets in high-resolution images. Summary of the Invention

[0005] The present invention provides an optimization processing method and device for a YOLO model applied to target detection, which can accurately identify small targets and edge targets in low-resolution images and accurately extract the structural information of targets in high-resolution images.

[0006] To solve the above technical problems, in a first aspect of the present invention, an optimization processing method for a YOLO model applied to target detection is disclosed, and the method includes:

[0007] Integrate an attention module into the target network of the YOLO model to be optimized, where the target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model, and the attention module includes a channel attention module and / or a spatial attention module;

[0008] Based on the attention module, perform average pooling operation and maximum pooling operation matching the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a maximum pooling result matching the attention module;

[0009] Generate an attention feature map of the target feature map based on the average pooling result and the max pooling result that match the attention module and the target feature map corresponding to the target network;

[0010] Among them, the attention feature map of the target feature map is used as the input feature map of the next layer network of the attention module.

[0011] As an optional implementation manner, in the first aspect of the present invention, when the attention module includes the channel attention module, performing an average pooling operation and a max pooling operation that match the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a max pooling result that match the attention module includes:

[0012] Performing an average pooling operation and a max pooling operation on each channel of the target feature map output from the target network based on the channel attention module to obtain the channel average pooling result and the channel max pooling result of each channel of the target feature map;

[0013] Among them, the generating an attention feature map of the target feature map based on the average pooling result and the max pooling result that match the attention module and the target feature map corresponding to the target network includes:

[0014] For any channel of the target feature map, determine the target weight of this channel according to the channel average pooling result and the channel max pooling result of this channel;

[0015] Generate a channel attention feature map of the target feature map according to the target weights of each channel of the target feature map and the target feature map output by the target network;

[0016] Among them, the determining the target weight of any channel of the target feature map according to the channel average pooling result and the channel max pooling result of this channel includes:

[0017] For any channel of the target feature map, perform an addition operation on the channel average pooling result and the channel max pooling result of this channel to obtain the channel sum pooling result of this channel;

[0018] Input the channel sum pooling result of this channel into a pre-determined first fully connected layer, and perform a non-linear transformation operation on the channel sum pooling result of this channel input into the first fully connected layer based on a pre-determined first activation function to obtain the transformation result of this channel;

[0019] Input the transformation result of the channel into a pre-determined second fully-connected layer, and perform a linear transformation operation on the transformation result of the channel input to the second fully-connected layer based on a pre-determined second activation function to obtain the target weight of the channel.

[0020] As an optional implementation manner, in the first aspect of the present invention, when the attention module includes the spatial attention module, the performing, on each channel of the target feature map output from the target network, an average pooling operation and a max pooling operation matching the attention module based on the attention module to obtain an average pooling result and a max pooling result matching the attention module includes:

[0021] Based on the spatial attention module, perform a channel average pooling operation and a channel max pooling operation on the spatial positions of each channel of the target feature map output from the target network to obtain a spatial average pooling result and a spatial max pooling result of the target feature map;

[0022] Wherein, the generating the attention feature map of the target feature map according to the average pooling result and the max pooling result matching the attention module and the target feature map corresponding to the target network includes:

[0023] Perform a concatenation operation on the spatial average pooling result and the spatial max pooling result of the target feature map to obtain a spatial concatenation feature result of the target feature map;

[0024] Generate the spatial attention feature map of the target feature map according to the concatenation feature result of the target feature map and the target feature map output from the target network;

[0025] Wherein, the generating the spatial attention feature map of the target feature map according to the concatenation feature result of the target feature map and the target feature map includes:

[0026] Based on a pre-determined first convolution kernel, perform a convolution operation on the concatenation feature result of the target feature map to obtain a convolution result of the target feature map, and based on a pre-determined third activation function, perform a normalization operation on the convolution result of the target feature map to obtain a corresponding normalization result of the target feature map; and generate the spatial attention feature map of the target feature map based on the corresponding normalization result of the target feature map and the target feature map output from the target network.

[0027] As an optional implementation manner, in the first aspect of the present invention, the method further includes:

[0028] Obtain the feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0029] For any two adjacent target layers, according to the attribute parameters of the feature maps of the two target layers, determine a second convolutional kernel that matches the attribute parameters, and based on the second convolutional kernel, perform a convolutional operation that matches the attribute parameters on the feature map of the target layer that matches the second convolutional kernel to obtain a convolutional feature map corresponding to the second convolutional kernel; wherein, the attribute parameters of the convolutional feature map corresponding to the second convolutional kernel are the same as the attribute parameters of the feature map of the other target layer among the two target layers except the target layer that matches the second convolutional kernel, and the attribute parameters include the number of channels and / or the spatial resolution;

[0030] Perform a feature fusion operation on the convolutional feature map corresponding to the second convolutional kernel and the feature map of the other target layer among the two target layers except the target layer that matches the second convolutional kernel to obtain a first multi-scale fusion feature map after fusion.

[0031] As an optional implementation manner, in the first aspect of the present invention, the method further includes:

[0032] Obtain the feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0033] For any three adjacent target layers, according to the resolution of the feature map of the target layer at the high level and the resolution of the feature map of the target layer at the middle level, determine a corresponding third convolutional kernel, and based on the third convolutional kernel, perform a resolution convolutional operation on the feature map of the target layer at the high level to obtain a transfer feature map of the target layer at the high level; according to the resolution of the feature map of the target layer at the low level and the resolution of the feature map of the target layer at the middle level, determine a corresponding fourth convolutional kernel, and based on the fourth convolutional kernel, perform a resolution convolutional operation on the feature map of the target layer at the low level to obtain a transfer feature map of the target layer at the low level, and the scales among the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level are equal;

[0034] Perform a channel dimension splicing operation on the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level to obtain a channel-spliced feature map, and based on a pre-determined fifth convolutional kernel, perform a convolutional operation on the channel-spliced feature map to obtain a second multi-scale fusion feature map.

[0035] As an optional implementation manner, in the first aspect of the present invention, the method further includes:

[0036] Determine the multiple performance parameters of each filter among the multiple filters of each of the multiple first network layers of the YOLO model and the scenario parameters of the application scenario of the YOLO model;

[0037] For any one of the filters, according to the scenario parameters of the application scenario, assign corresponding performance weights to each of the performance parameters of the filter, and calculate the importance of the filter according to each of the performance parameters and the performance weight of the performance parameter;

[0038] According to the importance of all the filters, delete the filters with importance less than or equal to the preset importance from the YOLO model to obtain the pruned YOLO model.

[0039] As an optional implementation manner, in the first aspect of the present invention, the multiple performance parameters of each filter include the loss change parameter of the YOLO model after cropping the filter, the L1 norm of the weight of the filter, and the scaling coefficient of the batch normalization of the filter;

[0040] Among them, the determination of the loss change parameter corresponding to each filter among the multiple filters of each of the multiple first network layers of the YOLO model includes:

[0041] Obtain the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of the filter;

[0042] According to the scenario parameters of the application scenario, respectively set corresponding loss change weights for the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of the filter;

[0043] Calculate the loss change parameter of the YOLO model after cropping the filter according to the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of the filter and the corresponding loss change weights.

[0044] As an optional implementation manner, in the first aspect of the present invention, the method further includes:

[0045] Determine all the second network layers that have performed filter pruning from all the first network layers in the pruned YOLO model;

[0046] For any one of the second network layers, determine the number of channels and the channel matrix of the second network layer, and update the number of channels and the channel matrix of the downstream network layer of the second network layer according to the number of channels and the channel matrix of the second network layer.

[0047] The second aspect of the present invention discloses an optimization processing device for a YOLO model applied to object detection, and the device includes:

[0048] An integration module, configured to integrate an attention module into a target network of a YOLO model to be optimized, where the target network of the YOLO model includes a backbone network of the YOLO model and / or a multi-scale fusion network of the YOLO model, and the attention module includes a channel attention module and / or a spatial attention module;

[0049] A pooling module, configured to perform an average pooling operation and a maximum pooling operation matching the attention module on each channel of a target feature map output from the target network based on the attention module, to obtain an average pooling result and a maximum pooling result matching the attention module;

[0050] A generation module, configured to generate an attention feature map of the target feature map according to the average pooling result and the maximum pooling result matching the attention module and the target feature map corresponding to the target network;

[0051] Wherein, the attention feature map of the target feature map is used as an input feature map of the next layer network of the attention module.

[0052] As an optional implementation manner, in the second aspect of the present invention, when the attention module includes the channel attention module, the specific manner in which the pooling module performs an average pooling operation and a maximum pooling operation matching the attention module on each channel of a target feature map output from the target network to obtain an average pooling result and a maximum pooling result matching the attention module includes:

[0053] Performing an average pooling operation and a maximum pooling operation on each channel of a target feature map output from the target network based on the channel attention module, to obtain a channel average pooling result and a channel maximum pooling result of each channel of the target feature map;

[0054] Wherein, the specific manner in which the generation module generates an attention feature map of the target feature map according to the average pooling result and the maximum pooling result matching the attention module and the target feature map corresponding to the target network includes:

[0055] For any channel of the target feature map, determining a target weight of the channel according to the channel average pooling result and the channel maximum pooling result of the channel;

[0056] Generating a channel attention feature map of the target feature map according to the target weights of each channel of the target feature map and the target feature map output from the target network;

[0057] Among them, for any channel of the target feature map, the specific manner in which the generation module determines the target weight of this channel according to the channel average pooling result and the channel maximum pooling result of this channel includes:

[0058] For any channel of the target feature map, perform an addition operation on the channel average pooling result and the channel maximum pooling result of this channel to obtain the channel sum pooling result of this channel;

[0059] Input the channel sum pooling result of this channel into a pre-determined first fully connected layer, and perform a non-linear transformation operation on the channel sum pooling result of this channel input to the first fully connected layer based on a pre-determined first activation function to obtain the transformation result of this channel;

[0060] Input the transformation result of this channel into a pre-determined second fully connected layer, and perform a linear transformation operation on the transformation result of this channel input to the second fully connected layer based on a pre-determined second activation function to obtain the target weight of this channel.

[0061] As an alternative implementation manner, in the second aspect of the present invention, when the attention module includes the spatial attention module, the specific manner in which the pooling module performs an average pooling operation and a maximum pooling operation matching the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a maximum pooling result matching the attention module includes:

[0062] Based on the spatial attention module, perform a channel average pooling operation and a channel maximum pooling operation on the spatial positions of each channel of the target feature map output from the target network to obtain the spatial average pooling result and the spatial maximum pooling result of the target feature map;

[0063] Among them, the specific manner in which the generation module generates the attention feature map of the target feature map according to the average pooling result and the maximum pooling result matching the attention module and the target feature map corresponding to the target network includes:

[0064] Perform a splicing operation on the spatial average pooling result and the spatial maximum pooling result of the target feature map to obtain the spatial splicing feature result of the target feature map;

[0065] Generate the spatial attention feature map of the target feature map according to the splicing feature result of the target feature map and the target feature map output from the target network;

[0066] Among them, the specific manner in which the generation module generates the spatial attention feature map of the target feature map according to the splicing feature result of the target feature map and the target feature map includes:

[0067] Performing a convolution operation on the splicing feature result of the target feature map based on a pre-determined first convolution kernel to obtain a convolution result of the target feature map, and performing a normalization operation on the convolution result of the target feature map based on a pre-determined third activation function to obtain a corresponding normalization result of the target feature map; and generating the spatial attention feature map of the target feature map based on the corresponding normalization result of the target feature map and the target feature map output by the target network.

[0068] As an alternative implementation manner, in the second aspect of the present invention, the device further includes:

[0069] An acquisition module, configured to acquire feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0070] A first determination module, configured to determine, for any two adjacent target layers, a second convolution kernel that matches the attribute parameter according to the attribute parameters of the feature maps of the two target layers;

[0071] A convolution module, configured to perform a convolution operation that matches the attribute parameter on the feature map of the target layer that matches the second convolution kernel based on the second convolution kernel to obtain a convolution feature map corresponding to the second convolution kernel; wherein, the attribute parameters of the convolution feature map corresponding to the second convolution kernel are the same as the attribute parameters of the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel, and the attribute parameters include the number of channels and / or the spatial resolution;

[0072] A fusion module, configured to perform a feature fusion operation on the convolution feature map corresponding to the second convolution kernel and the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel to obtain a first multi-scale fusion feature map after fusion.

[0073] As an alternative implementation manner, in the second aspect of the present invention, the device further includes:

[0074] An acquisition module, configured to acquire feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0075] A first determination module, configured to determine a corresponding third convolution kernel according to the resolution of the feature map of the target layer at the high layer and the resolution of the feature map of the target layer at the middle layer for any three adjacent target layers;

[0076] A convolution module, configured to perform a resolution convolution operation on the feature map of the target layer at a high level based on the third convolution kernel, to obtain a transfer feature map of the target layer at a high level;

[0077] The first determination module is further configured to determine a corresponding fourth convolution kernel according to the resolution of the feature map of the target layer at a low level and the resolution of the feature map of the target layer at a middle level;

[0078] The convolution module is further configured to perform a resolution convolution operation on the feature map of the target layer at a low level based on the fourth convolution kernel, to obtain a transfer feature map of the target layer at a low level, and the scales of the transfer feature map of the target layer at a low level, the transfer feature map of the target layer at a high level, and the feature map of the target layer at a middle level are equal;

[0079] A splicing module, configured to perform a channel dimension splicing operation on the transfer feature map of the target layer at a low level, the transfer feature map of the target layer at a high level, and the feature map of the target layer at a middle level, to obtain a channel-spliced feature map;

[0080] The convolution module is further configured to perform a convolution operation on the channel-spliced feature map based on a pre-determined fifth convolution kernel, to obtain a second multi-scale fusion feature map.

[0081] As an optional implementation manner, in the second aspect of the present invention, the apparatus further includes:

[0082] A second determination module, configured to determine multiple performance parameters of each filter among multiple filters of each first network layer of the YOLO model and scene parameters of the application scenario of the YOLO model;

[0083] An allocation module, configured to allocate corresponding performance weights to each performance parameter of the filter according to the scene parameters of the application scenario for any one of the filters;

[0084] A calculation module, configured to calculate the importance of the filter according to each performance parameter and the performance weight of the performance parameter;

[0085] A deletion module, configured to delete filters with importance less than or equal to a preset importance from the YOLO model according to the importance of all the filters, to obtain the pruned YOLO model.

[0086] As an optional implementation manner, in the second aspect of the present invention, the multiple performance parameters of each filter include a loss change parameter of the YOLO model after cropping the filter, an L1 norm of the weight of the filter, and a scaling coefficient of batch normalization of the filter;

[0087] Among them, the specific manner in which the second determination module determines the loss change parameter corresponding to each filter among the multiple filters of each first network layer of the YOLO model includes:

[0088] Obtain the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of this filter;

[0089] According to the scene parameters of the application scenario, set corresponding loss change weights for the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of this filter respectively;

[0090] Calculate the loss change parameter of the YOLO model after pruning this filter according to the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of this filter and the corresponding loss change weight.

[0091] As an optional implementation manner, in the second aspect of the present invention, the second determination module is further configured to determine, from all the first network layers in the pruned YOLO model, all the second network layers that have performed filter pruning;

[0092] The second determination module is further configured to determine the number of channels and the channel matrix of any one of the second network layers;

[0093] The device further includes:

[0094] An update module, configured to update the number of channels and the channel matrix of the downstream network layer of the second network layer according to the number of channels and the channel matrix of the second network layer.

[0095] The third aspect of the present invention discloses another optimization processing device for a YOLO model applied to target detection, and the device includes:

[0096] A memory storing executable program code;

[0097] A processor coupled to the memory;

[0098] The processor calls the executable program code stored in the memory and executes some or all of the steps in any one of the optimization processing methods for a YOLO model applied to target detection disclosed in the first aspect of the present invention.

[0099] The sixth aspect of the present invention discloses a computer storage medium storing computer instructions, which are used to execute some or all steps of any one of the optimization processing methods of the YOLO model applied to target detection disclosed in the first aspect of the present invention when called.

[0100] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0101] In the embodiments of the present invention, an attention module is incorporated into the target network of the YOLO model to be optimized. The target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model. The attention module includes a channel attention module and / or a spatial attention module. Based on the attention module, average pooling operation and max pooling operation matching the attention module are performed on each channel of the target feature map output from the target network to obtain an average pooling result and a max pooling result matching the attention module. An attention feature map of the target feature map is generated according to the average pooling result and the max pooling result matching the attention module and the target feature map corresponding to the target network. Among them, the attention feature map of the target feature map is used as the input feature map of the next layer network of the attention module. It can be seen that by incorporating the attention module into the backbone network and / or the multi-scale fusion network of the YOLO model, the present invention optimizes the YOLO model, and can obtain a YOLO model that emphasizes important feature regions in the image, so that when the YOLO model performs target recognition in the image, it can accurately recognize small targets and edge targets in low-resolution images, and can accurately extract the structural information of targets in high-resolution images, thereby facilitating improving the recognition accuracy and efficiency of various size targets in the image and meeting the requirements of fine inspection tasks. Moreover, the optimized YOLO model not only has high precision, but also has lightweight characteristics and can be flexibly deployed on embedded devices and servers to meet the target recognition requirements of various scenarios. Description of the Drawings

[0102] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for description in the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0103] Figure 1 It is a schematic flowchart of an optimization processing method of a YOLO model applied to target detection disclosed in an embodiment of the present invention;

[0104] Figure 2 It is a schematic flowchart of another optimization processing method of a YOLO model applied to target detection disclosed in an embodiment of the present invention;

[0105] Figure 3 It is a schematic structural diagram of an optimization processing device for a YOLO model applied to object detection disclosed in an embodiment of the present invention;

[0106] Figure 4 It is a schematic structural diagram of another optimization processing device for a YOLO model applied to object detection disclosed in an embodiment of the present invention;

[0107] Figure 5 It is a schematic structural diagram of yet another optimization processing device for a YOLO model applied to object detection disclosed in an embodiment of the present invention. Detailed implementation manners

[0108] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0109] The terms "first", "second", etc. in the specification and claims of the present invention and the above drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or terminal that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or terminals.

[0110] Referring to "embodiments" herein means that a specific feature, structure, or characteristic described in connection with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0111] The present invention discloses an optimization processing method and device for a YOLO model applied to object detection. By integrating an attention module into the backbone network and / or multi-scale fusion network of the YOLO model, the YOLO model is optimized, and a YOLO model that emphasizes important feature regions in an image can be obtained. When the YOLO model performs object recognition in an image, it can accurately recognize small objects and edge objects in a low-resolution image, and can also accurately extract the structural information of the objects in a high-resolution image, thereby facilitating the improvement of the recognition accuracy and efficiency of various-sized objects in the image and meeting the requirements of fine patrol tasks. Moreover, the optimized YOLO model not only has high precision but also is lightweight and can be flexibly deployed on embedded devices and servers to meet the object recognition requirements of various scenarios. The following will be described in detail respectively.

[0112] Embodiment 1

[0113] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of an optimization processing method for a YOLO model applied to object detection disclosed in an embodiment of the present invention. Among them, Figure 1 the described method can be applied to any scenario that requires object detection. Among them, the scenario includes but is not limited to border patrol scenarios, power inspection scenarios, and forest fire prevention scenarios. And the scenario is provided with a corresponding flying device. Optionally, the flying device is performing a corresponding flying task in the corresponding scenario. Among them, as Figure 1 shown, the method may include the following operations:

[0114] 101. Integrate the attention module into the target network of the YOLO model to be optimized. The target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model. The attention module includes a channel attention module and / or a spatial attention module.

[0115] In an embodiment of the present invention, optionally, integrating the attention module into the target network of the YOLO model can be understood as integrating it between multiple network layers in the target network and / or into the last network layer of the target network. The YOLO model includes but is not limited to one of YOLO3, YOLO4, YOLO5, YOLO6, and YOLO8.

[0116] 102. Perform an average pooling operation and a maximum pooling operation that match the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a maximum pooling result that match the attention module.

[0117] 103. Generate an attention feature map of the target feature map based on the average pooling result and the maximum pooling result that match the attention module and the target feature map corresponding to the target network; wherein, the attention feature map of the target feature map is used as the input feature map of the next layer network of the attention module.

[0118] In an embodiment of the present invention, optionally, the next layer network of the attention module can be the last network layer of the target network of the YOLO model or an intermediate network layer of the target network.

[0119] It can be seen that implementing Figure 1 The described method optimizes the YOLO model by integrating the attention module into the backbone network and / or the multi-scale fusion network of the YOLO model, and can obtain a YOLO model that emphasizes important feature regions in the image, so that when the YOLO model performs target recognition in the image, it can accurately recognize small targets and edge targets in low-resolution images, and can accurately extract the structural information of targets in high-resolution images, thereby facilitating improving the recognition accuracy and efficiency of various-sized targets in the image and meeting the requirements of the refined inspection task; moreover, the optimized YOLO model not only has high accuracy but also is lightweight, and can be flexibly deployed on embedded devices and servers to meet the target recognition needs of various scenarios.

[0120] In an embodiment of the present invention, optionally, when the attention module includes a channel attention module, perform an average pooling operation and a maximum pooling operation that match the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a maximum pooling result that match the attention module, including:

[0121] Perform an average pooling operation and a maximum pooling operation on each channel of the target feature map output from the target network based on the channel attention module to obtain the channel average pooling result and the channel maximum pooling result of each channel of the target feature map;

[0122] Wherein, generating an attention feature map of the target feature map based on the average pooling result and the maximum pooling result that match the attention module and the target feature map corresponding to the target network includes:

[0123] For any channel of the target feature map, determine the target weight of this channel according to the channel average pooling result and the channel maximum pooling result of this channel;

[0124] Generate a channel attention feature map of the target feature map according to the target weights of each channel of the target feature map and the target feature map output from the target network;

[0125] Among them, for any one of the channels of the target feature map, determining the target weight of this channel according to the channel average pooling result and the channel maximum pooling result of this channel includes:

[0126] For any one channel of the target feature map, perform an addition operation on the channel average pooling result and the channel

[0127] maximum pooling result of this channel to obtain the channel sum pooling result of this channel;

[0128] Input the channel sum pooling result of this channel into a pre-determined first fully connected layer, and perform a non-linear transformation operation on the channel sum pooling result of this channel input to the first fully connected layer based on a pre-determined first activation function to obtain the transformation result of this channel;

[0129] Input the transformation result of this channel into a pre-determined second fully connected layer, and perform a linear transformation operation on the transformation result of this channel input to the second fully connected layer based on a pre-determined second activation function to obtain the target weight of this channel.

[0130] In an embodiment of the present invention, optionally, the size of the target feature map F is C×H×W, where C, H, and W are the number of channels, height, and width of the target feature map F respectively.

[0131] For each channel of the target feature map F, the calculation formulas for the channel average pooling result and the channel maximum pooling result of this channel are respectively:

[0132]

[0133] In the formula, G arg1 (F), G max1 (F) are respectively the channel average pooling result of any one channel of the target feature map F and the channel maximum pooling result of this channel, and F(c, i, j) is the pixel coordinates, that is, the spatial position, of the i-th and j-th of the target feature map F in this channel. Among them, the channel maximum pooling result and the channel average pooling result of this channel are both a C×1×1 vector.

[0134] For any one channel of the target feature map F, the calculation formulas for the transformation result and the target weight of this channel are as follows respectively:

[0135] X0 = ReLU(W0(G avg1 (F) + G max1 (F)));

[0136] M c (F) = σ1(W1X0);

[0137] In the formula, X0 is the transformation result of any one channel of the target feature map F; M c(F) is the target weight of the channel; W0 and W1 are the corresponding first fully connected layer and second fully connected layer respectively; ReLU() and σ1() are the corresponding first activation function and second activation function respectively. Optionally, ReLU() can also be replaced with the Leaky ReLU activation function, or the Mish or SiLU (Swish) activation function, so as to further improve the non-linear expression ability and training stability of the YOLO model. Among them, the Mish activation function is smooth and continuous, has better gradient propagation properties, helps to reduce the situation of gradient disappearance, can better capture complex features, and is especially suitable for the scenarios with complex backgrounds in UAV target detection.

[0138] In the embodiments of the present invention, the output value of the target weight of each channel is within the range of [0,1]. Among them, the larger the output value, the more important the feature representation of the corresponding channel in the target feature map F is, that is, the greater the contribution to the task.

[0139] In the embodiments of the present invention, after obtaining the target weight of each channel of the target feature map F, a vector of the target weight of the channel is generated, such as with a length of C.

[0140] The calculation formula of the channel attention feature map of the target feature map is as follows:

[0141] F out1 =M c (F)⊙F;

[0142] In the formula, F out1 is the channel attention feature map of the target feature map; ⊙ represents element-wise multiplication, that is, the target weight of each channel is multiplied element-wise with the corresponding pixel coordinates (spatial positions) in the target feature map F.

[0143] It can be seen that the embodiments of the present invention can also extract important channel information by performing global average pooling and global max pooling on each channel of the feature map, and generate corresponding channel weights through a shared fully connected layer and an activation function, that is, weight the feature map from the channel dimension, enhance the importance of the feature map representation by assigning different weights to different channels, so as to emphasize the channels in the input feature map that make important contributions to the target task in the YOLO model, and suppress irrelevant or unimportant channels, thereby improving the expression of important features and improving the recognition accuracy and reliability of the YOLO model for targets of various sizes.

[0144] In the embodiments of the present invention, as an optional implementation manner, when the attention module includes a spatial attention module, for each channel of the target feature map output from the target network based on the attention module, perform an average pooling operation and a max pooling operation matching the attention module to obtain an average pooling result and a max pooling result matching the attention module, including:

[0145] Based on the spatial attention module, perform channel average pooling operation and channel maximum pooling operation on the spatial positions of each channel of the target feature map output from the target network to obtain the spatial average pooling result and the spatial maximum pooling result of the target feature map;

[0146] Among them, according to the average pooling result and the maximum pooling result matching the attention module and the target feature map corresponding to the target network, generate the attention feature map of the target feature map, including:

[0147] Perform a concatenation operation on the spatial average pooling result and the spatial maximum pooling result of the target feature map to obtain the spatial concatenation feature result of the target feature map;

[0148] According to the concatenation feature result of the target feature map and the target feature map output by the target network, generate the spatial attention feature map of the target feature map;

[0149] Among them, according to the concatenation feature result of the target feature map and the target feature map, generate the spatial attention feature map of the target feature map, including:

[0150] Based on the pre-determined first convolution kernel, perform a convolution operation on the concatenation feature result of the target feature map to obtain the convolution result of the target feature map, and based on the pre-determined third activation function, perform a normalization operation on the convolution result of the target feature map to obtain the corresponding normalization result of the target feature map; and based on the corresponding normalization result of the target feature map and the target feature map output by the target network, generate the spatial attention feature map of the target feature map.

[0151] In the embodiment of the present invention, optionally, optionally, the size of the target feature map F is C×H×W, where C, H, and W are the number of channels, height, and width of the target feature map F respectively. Perform channel dimension average pooling and maximum pooling on all spatial positions of the target feature map F through the following formulas respectively to obtain the spatial average pooling result and the spatial maximum pooling result of the target feature map F. Among them, the calculation formulas for channel dimension average pooling and channel dimension maximum pooling are respectively:

[0152] where i∈[1,H],j∈[1,W];

[0153] where i∈[1,II],:∈[1,W];

[0154] In the formula, G arg2 (F) and G max2 (F) are respectively the spatial average pooling result and the spatial maximum pooling result of the target feature map F, and both are feature maps of 1×H×W.

[0155] Optionally, the calculation formula for the spatial splicing feature result of the target feature map is as follows:

[0156] G(F) = Concat[G arg2 (F), G max2 (F)];

[0157] In the formula, G(F) is the spatial splicing feature result of the target feature map, which is a 2×H×W tensor; Concat represents the splicing operation in the channel dimension.

[0158] Optionally, the calculation formulas for the normalization result corresponding to the target feature map and the spatial attention feature map of the target feature map are as follows:

[0159] M s (F) = σ2(Conv(G(G)));

[0160] F out2 = M s (F) ⊙ F;

[0161] In the formula, M s (F) and F out2 are respectively the normalization result corresponding to the target feature map F and the spatial attention feature map of the target feature map F; Conv represents the convolution operation; σ2() is the third activation function, such as Sigmoid, which is used to normalize the output to the range of [0,1]; ⊙ represents element-wise multiplication, that is, multiplying the attention weight at each spatial position by the pixel value of the corresponding channel.

[0162] It can be seen that the embodiments of the present invention can also use global average pooling and global max pooling methods to perform channel dimension average pooling and channel dimension max pooling on all spatial positions of each channel, so as to compress the feature map in the channel dimension and generate the weight of each spatial position, so that the YOLO model can pay more attention to the regions contributing to the task. This mechanism compresses the high-dimensional information of the feature map and converts it into two-dimensional features, thereby generating a spatial-level attention map to strengthen the features at spatial positions; and combined with the aforementioned channel attention weighting, it enables the YOLO model to more accurately focus on the target region, suppress irrelevant regions and unimportant features. In tasks such as multi-scale and small target detection, it can significantly improve the performance of the YOLO model, enhance the attention and expression of key features, ultimately improve the accuracy and robustness of the YOLO model, and further improve the recognition accuracy of various size targets and multi-targets, as well as improve the adaptability of various application scenarios.

[0163] In an optional embodiment, the method may further include the following steps:

[0164] Obtain the feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0165] For any two adjacent target layers, according to the attribute parameters of the feature maps of the two target layers, determine a second convolution kernel that matches the attribute parameters, and based on the second convolution kernel, perform a convolution operation that matches the attribute parameters on the feature map of the target layer that matches the second convolution kernel to obtain a convolution feature map corresponding to the second convolution kernel; wherein, the attribute parameters of the convolution feature map corresponding to the second convolution kernel are the same as the attribute parameters of the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel, and the attribute parameters include the number of channels and / or the spatial resolution;

[0166] Perform a feature fusion operation on the convolution feature map corresponding to the second convolution kernel and the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel to obtain a first multi-scale fusion feature map after fusion.

[0167] In this optional embodiment, optionally, when the attribute parameter includes the number of channels, if the number of channels between the two target layers is different, the number of channels of the target layer with more channels can be adjusted to be the same as the number of channels of the target layer with fewer channels, or the number of channels of the target layer with fewer channels can be adjusted to be the same as the number of channels of the target layer with more channels, so that the number of channels of the two target layers is the same. For example, if the bottom feature map of the two target layers has 64 channels and the high-level feature map has 256 channels, the number of channels of the low-level feature map can be adjusted from 64 to 256 through a 1×1 convolution kernel, so that the two target layers have the same number of channels and improve the accuracy of feature fusion.

[0168] When the attribute parameter includes the spatial resolution, if the spatial resolutions between the two target layers are different, the spatial resolution of the target layer with a high spatial resolution can be downsampled to be the same as the spatial resolution of the target layer with a low spatial resolution, or the spatial resolution of the target layer with a low spatial resolution can be upsampled to be the same as the spatial resolution of the target layer with a high spatial resolution, so that the spatial resolutions of the two target layers are the same. For example, if the spatial resolution of the bottom feature map of the two target layers is 128×128 and the spatial resolution of the high-level feature map is 32×32, the spatial resolution of the low-level feature map can be downsampled (such as using pooling or convolution operations) to 32×32 through a 3×3 convolution kernel or a 5×5 convolution kernel, or the spatial resolution of the high-level feature map can be upsampled (such as using transposed convolution or bilinear interpolation) to 128×128 through a 3×3 convolution kernel plus a stride or a 5×5 convolution plus a stride, so that the two target layers have the same spatial resolution and improve the accuracy of feature fusion.

[0169] Optionally, the fusion formula of the feature maps of the two target layers is as follows:

[0170]

[0171] In the formula, F out3 represents the first multi-scale fusion feature map after fusion, and F low represents the low-level feature map in the features of two target layers, including details and spatial information; F high represents the high-level feature map in the features of two target layers, including semantic and context information; Conv1 and Conv2 respectively represent the convolution operations for channel transformation and downsampling / upsampling of the features of the two target layer feature maps; represents pixel-wise addition, which is used to fuse the features of two target layers.

[0172] In this optional embodiment, optionally, the feature maps output by different target layers in the multi-scale fusion network may be derived from the aforementioned spatial attention feature map or channel attention feature map or the feature map output by the backbone network.

[0173] It can be seen that the embodiments of the present invention can not only transfer the high-level features with rich semantic information to the low-level features with high spatial resolution, but also gradually transfer the low-level features with rich detail information to the high-level features with rich semantic information by simultaneously using the FPN method of top-down fusion and bottom-up fusion, enabling the feature information of different scales to interact more closely, realizing the full fusion of multi-scale features, significantly improving the multi-scale detection performance of the YOLO model, further enhancing the recognition accuracy of the YOLO model in small target and multi-target scenarios, and meeting the refined target detection requirements.

[0174] In another optional implementation manner, the method may further include the following steps:

[0175] Obtain the feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0176] For any three adjacent target layers, determine the corresponding third convolution kernel according to the resolution of the feature map of the target layer at the high level and the resolution of the feature map of the target layer at the middle level, and based on this third convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the high level to obtain the transfer feature map of the target layer at the high level; determine the corresponding fourth convolution kernel according to the resolution of the feature map of the target layer at the low level and the resolution of the feature map of the target layer at the middle level, and based on this fourth convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the low level to obtain the transfer feature map of the target layer at the low level, and the scales of the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level are equal;

[0177] Perform a channel dimension concatenation operation on the transfer feature map of the target layer at the low layer, the transfer feature map of the target layer at the high layer, and the feature map of the target layer at the middle layer to obtain a channel concatenated feature map, and perform a convolution operation on the channel concatenated feature map based on a pre-determined fifth convolution kernel to obtain a second multi-scale fusion feature map.

[0178] In this optional embodiment, optionally, the second multi-scale fusion feature map is calculated by the following formula:

[0179] F out4 = Conv4(Concat(F up , F down , F horizontal ));

[0180] In the formula, F out4 represents the second multi-scale fusion feature map; F up represents the transfer feature map of the target layer at the high layer; F down represents the transfer feature map of the target layer at the low layer; F horizontal represents the feature map of the target layer at the middle layer; Concat represents the channel dimension concatenation operation; Conv4 represents the convolution operation after fusing the transfer feature map of the target layer at the high layer, the transfer feature map of the target layer at the low layer, and the feature of the target layer at the middle layer. Figure 3 of the map.

[0181] It can be seen that in this optional embodiment, by adding a lateral connection in the up and down transfer paths of the FPN, directly transmitting information between features of the same scale across layers, strengthening the multi-directional smooth interaction of each layer of features, thereby enhancing the feature information sharing ability, and through the lateral connection and convolution operation, each layer of features contains richer context information, enhancing the ability of the YOLO model to capture details, especially more effective for the detection of small-sized targets, and at the same time reducing the detection misses caused by the difference in target sizes, improving the detection effect for small-sized targets and complex scenes, especially suitable for complex scenes with small and numerous target areas, such as drone scenes.

[0182] In yet another optional embodiment, the method may further include the following steps:

[0183] Determine the multiple performance parameters of each filter in the multiple filters of each first network layer of the YOLO model and the scene parameters of the application scenario of the YOLO model;

[0184] For any filter, according to the scene parameters of the application scenario, assign corresponding performance weights to each performance parameter of the filter, and calculate the importance of the filter according to each performance parameter and the performance weight of the performance parameter;

[0185] According to the importance of all filters, filters with importance less than or equal to the preset importance are deleted from the YOLO model to obtain the pruned YOLO model.

[0186] In this optional embodiment, optionally, multiple performance parameters of each filter include the loss change parameter of the YOLO model after cropping the filter, the L1 norm of the filter, and the scaling coefficient of batch normalization of the filter. Optionally, for scenario parameters in different application scenarios, each performance parameter corresponds to a different performance weight. Further, through cross-validation or model performance evaluation experiments, the best combination of performance weights can be determined according to the specific application scenario to ensure that scores in different dimensions can reasonably reflect the importance of the filter. For example, for application scenarios more sensitive to the balance of feature distribution (such as object detection or segmentation), increase the performance weight of the scaling coefficient of batch normalization, and control the priority of sensitivity analysis by adjusting the performance weight of the loss change parameter of the YOLO model after cropping the filter; if more attention is paid to the prediction accuracy of the pruned model, give a higher performance weight to the loss change parameter of the YOLO model after cropping the filter; in image classification, the performance weight of the L1 norm of the weight of the filter can be appropriately increased to improve the sensitivity to the weight intensity. Among them, the sum of the performance weights corresponding to all performance parameters is equal to 1.

[0187] Importance(w i )=a·||w i ||1+β·|Y i |+γ·ΔE(w i );

[0188]

[0189] In the formula, Importance(w i ) represents the importance of the i-th filter in the YOLO model; ||w i ||1 represents the L1 norm of the weight of the i-th filter, that is, it represents the weight intensity of filter i. The larger the L1 norm, the more significant the contribution of the filter in the network; |γ i | represents the scaling coefficient of the corresponding i-th filter in the BatchNorm layer (batch normalization layer) of the YOLO model, reflecting the adjustment ability of the i-th filter to the feature distribution. The larger the |γ i | value, the more important the influence of the filter on the performance of the YOLO model; ΔE(w i) represents the change in the output error of the YOLO model after cropping the i-th filter, which is used to quantify the actual impact of this filter on the performance of the YOLO model. By calculating the output difference of the model after cropping a certain filter, its importance can be more accurately evaluated; α, β, and γ are the performance weights of the corresponding performance parameters; w i，j，k，c represents the weight of the j-th row, k-th column, and c-th input channel of the i-th filter. Specifically, j and k represent the spatial dimensions of the convolutional kernel (i.e., the height and width of the convolutional kernel), and c represents the index of the input channel. By summing up the weights of all filters using the summation formula, the total L1 norm of the filter is obtained by traversing the weights of all spatial positions and input channels.

[0190] In this optional embodiment, optionally, after obtaining the importance of each filter, it can be sorted or not sorted, and the filters with importance less than or equal to the preset importance are deleted. Further, the preset importance can be determined according to the hardware performance of the device where the YOLO model is integrated. Among them, the hardware performance includes computing power and / or memory / storage capacity. The stronger the computing power and the stronger the memory / storage capacity, the lower the preset importance, that is, the lower the pruning rate of the YOLO model. For example, high-performance hardware with high computing power can support a lower pruning rate (30%), maintaining high accuracy; low-performance hardware with high computing power (such as embedded devices) requires a higher pruning rate (50%) to reduce the amount of computation; hardware with limited memory requires a higher pruning rate (50%) to reduce memory occupancy; hardware with low latency requirements (such as real-time systems) requires a higher pruning rate to accelerate the inference process; high-performance processors are suitable for a 30% pruning rate, while edge devices require a 50% pruning rate.

[0191] It can be seen that this optional embodiment analyzes the importance of the filter by combining the loss change parameters of the YOLO model after cropping the corresponding filter, the L1 norm of the corresponding filter, the scaling coefficient of batch normalization of the corresponding filter, the corresponding performance weights, and the sensitivity analysis of the YOLO model output, improving the accuracy and reliability of determining the importance of each filter, screening and removing the filters (channels) that have less impact on the final output of the YOLO model, thereby improving the pruning accuracy of the YOLO model, reducing the adverse impact on the performance of the YOLO model during the pruning process, significantly reducing the number of parameters and the amount of computation of the YOLO model, reducing memory occupancy and inference latency, and improving the inference speed of the YOLO model while maintaining the prediction accuracy of the YOLO model, which is particularly suitable for low-power computing environments such as embedded devices and edge computing scenarios.

[0192] In this optional embodiment, determining the loss change parameter corresponding to each filter among the multiple filters of each first network layer of the YOLO model includes:

[0193] Obtain the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter;

[0194] According to the scenario parameters of the application scenario, set the corresponding loss change weights for the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter respectively;

[0195] Calculate the loss change parameter of the YOLO model after pruning the filter according to the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter and the corresponding loss change weights.

[0196] In this optional embodiment, optionally, for any filter of the YOLO model, the calculation formula of the loss change parameter of the YOLO model after pruning the filter can be as follows:

[0197]

[0198] In the formula, and respectively represent the loss parameters of the YOLO model after pruning the i-th filter and the original YOLO model.

[0199] In this optional embodiment, further optionally, for multi-task scenarios such as object detection, ΔE(w i ) can be further divided into:

[0200] ΔE(w i ) = λ cls ·ΔL cls + λ bbox ·ΔL bbox + λ d ·ΔL d ;

[0201] In the formula, ΔL cls represents the classification loss change parameter of the i-th filter; ΔL bbox represents the bounding box regression loss change parameter of the i-th filter; ΔL d represents the confidence loss change parameter of the i-th filter; λ cls , λ d , λ bbox respectively represent the corresponding loss change weights, which are used to adjust the importance weights of different task losses for pruning.

[0202] In this optional embodiment, optionally, when calculating ΔE(w i) When it is possible, the contribution degree of each filter to the loss function can be obtained through gradient backpropagation, so as to more accurately evaluate the importance of the filter; and output sensitivity analysis ensures that the filters sensitive to the performance of the model task are preferentially retained, thus effectively reducing the damage of pruning to the accuracy of the final model while optimizing the inference efficiency.

[0203] It can be seen that this optional embodiment can also analyze the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter and their corresponding loss change weights, which can improve the determination accuracy of the loss change parameter of the filter, thus facilitating further improvement of the determination accuracy and reliability of the importance of the filter, and further lightweighting the YOLO model while ensuring the YOLO model, which is conducive to more flexible deployment on lightweight embedded devices.

[0204] In yet another optional embodiment, the method may further include the following steps:

[0205] From the pruned YOLO model, among all the first network layers, determine all the second network layers that have performed filter pruning;

[0206] For any second network layer, determine the number of channels and the channel matrix of the second network layer, and update the number of channels and the channel matrix of the downstream network layer of the second network layer according to the number of channels and the channel matrix of the second network layer.

[0207] In this optional embodiment, optionally, determine the number of output channels after pruning. After removing unimportant filters, the number of output channels of the convolutional layer decreases. For example, if the original number of output channels is C and k channels are removed, the number of output channels becomes C - k. At this time, it is necessary to update the number of input channels of the downstream network, that is, the number of input channels of the subsequent convolutional layer needs to be adjusted to the number of output channels of the pruned convolutional layer. For example, the original number of input channels of the downstream layer is C and it is adjusted to C - k after pruning. And adjust the convolutional layer weights: after pruning, the shape of the weight matrix of the convolutional layer needs to be updated to remove the corresponding channels.

[0208] It can be seen that after pruning the YOLO model in this optional embodiment, the number of input channels and the channel matrix of the downstream network are further adjusted to ensure the consistency of the structure of the YOLO model, thereby ensuring the usage accuracy of the model.

[0209] In yet another optional embodiment, the method may further include the following steps:

[0210] After updating the number of channels and the channel matrix of the downstream network layer of the second network layer, obtain a feature map data set, and based on this feature data set, perform a training operation on the YOLO model to obtain a trained YOLO model.

[0211] In this alternative embodiment, the trained YOLO model is integrated into an embedded device and a server, and the embedded device is integrated onto a flying device. Among them, the embedded device is used to collect images and, together with the server, perform object detection on the collected images based on the YOLO model.

[0212] In this alternative embodiment, optionally, the feature map dataset is composed of feature maps of different scenarios and includes various objects of different sizes.

[0213] It can be seen that after pruning and structure adjustment of the YOLO model in this alternative embodiment, it is further fine-tuned, enabling the model to further improve the balance among the recognition accuracy, reliability, and recognition efficiency of different-sized objects and multiple objects while maintaining high accuracy, especially suitable for application scenarios with resource constraints and high real-time requirements, such as flight mission scenarios.

[0214] In yet another alternative embodiment, the method may further include the following steps:

[0215] The server obtains a first image sent by an embedded device communicatively connected to it. The first image is an image collected by the embedded device for the current scenario during the flight of the flying device, and the first image contains an object first recognized by the embedded device.

[0216] The server performs a detection operation on the first image based on the YOLO model to obtain the detection result of the first image. Among them, the object network of the YOLO model incorporates an attention module, where the attention module includes a spatial attention module and / or a channel attention module. The attention module is a module determined for each channel of the feature map output by the object network of the YOLO model. The object network of the YOLO model includes the backbone network and / or the multi-scale fusion network of the YOLO model.

[0217] The server identifies the information of at least one first object in the first image according to the detection result of the first image. All first objects include all objects first recognized, and the information of each first object includes the category of the first object.

[0218] In the embodiments of the present invention, further, the information of each first object further includes at least one of color, size, and position in the current scenario.

[0219] It can be seen that in this alternative embodiment, the YOLO model integrating the backbone network and / or the multi-scale fusion network into the attention module is integrated on the server side, and it is used to perform object detection operations on the images of newly recognized targets first identified from the embedded device. It can not only accurately identify small targets and edge targets in low-resolution images, but also accurately extract the structural information of the targets in high-resolution images, improving the recognition efficiency and accuracy of multiple targets; and the collaborative processing of first performing object detection and capture through the embedded device and then performing refined recognition through the server improves the image data processing efficiency and balances the requirements of object detection efficiency and accuracy, especially suitable for the inspection scenario of high-resolution images.

[0220] In yet another alternative embodiment, before the server performs a detection operation on the first image based on the YOLO model to obtain the detection result of the first image, the method may further include the following steps:

[0221] The server performs an image chunking operation on the obtained first image to obtain a plurality of chunk images;

[0222] Wherein, the server performs a detection operation on the first image based on the YOLO model to obtain the detection result of the first image, including:

[0223] The server performs a detection operation on each chunk image based on the YOLO model to obtain the detection result of each chunk image;

[0224] Wherein, the server identifies the information of at least one first target in the first image according to the detection result of the first image, including:

[0225] The server identifies the information of at least one first target in the first image according to the detection results of all the chunk images.

[0226] It can be seen that in this alternative embodiment, by first chunking the images obtained from the embedded device and then sequentially using the above-mentioned YOLO model to perform object detection and recognition on the chunked images, the efficiency and accuracy of object detection and recognition are improved.

[0227] In this alternative embodiment, as an alternative implementation manner, the detection result of each chunk image includes a plurality of detection boxes, the confidence level and category of each detection box;

[0228] Wherein, the server identifies the information of at least one first target in the first image according to the detection results of all the chunk images, including:

[0229] The server performs a splicing operation on the detection results of each chunk image to obtain the spliced first detection result;

[0230] The server determines, from all the detection boxes in the first detection result, all detection box groups with an overlap degree greater than or equal to a preset overlap degree according to the coordinates of each detection box in the first detection result, and each detection box group consists of at least two detection boxes;

[0231] The server deletes all detection boxes except the detection box with the highest confidence among all the detection boxes in the detection box group according to the confidence of each detection box in each detection box group;

[0232] After performing the detection box deletion operation on all detection box groups, the server performs a splicing operation on adjacent boxes in the first detection result to obtain a second detection result, and identifies information about at least one first target in the first image according to the second detection result.

[0233] In this optional embodiment, optionally, the preset overlap degree can be determined according to the type of the current scene, that is, for different scenes, the corresponding preset overlap degrees are different.

[0234] In this optional embodiment, optionally, the detection box deletion can be performed in any way, such as the non-maximum suppression method.

[0235] It can be seen that in this optional embodiment, splicing is performed by optimizing the target detection results of the block images to ensure the accuracy and integrity of the detection boxes. After splicing, grouping is performed according to the overlap degree between the detection boxes. For each group of detection boxes, the detection box with the highest confidence is retained, and other detection boxes are deleted, and then adjacent detection boxes are spliced, which improves the detection accuracy and data volume after splicing, thereby improving the detection efficiency and accuracy of the target.

[0236] In another optional embodiment, the method may further include the following steps:

[0237] The server determines the number of blocks into which the first image needs to be divided, and obtains the image size of the first image and the size conditions of all first targets in the first image;

[0238] The server determines the first step length in the horizontal direction and the second step length in the vertical direction according to the size conditions of all first targets in the first image;

[0239] The server determines the block overlap degree during the process of dividing the first image into blocks according to the first step length, the second step length, the image size of the first image, and the number of blocks;

[0240] Among them, the server performs an image block division operation on the received first image to obtain a plurality of block images, including:

[0241] The server performs an image block division operation on the received first image according to the block overlap degree to obtain a plurality of block images.

[0242] In this optional embodiment, optionally, the number of blocks for chunking is determined by at least one of the resolution of the first image, the image size, and the type of the current scene. The lower the resolution, the larger the size, and the higher the requirement for the target detection accuracy indicated by the type of the current scene, the more the number of blocks for chunking.

[0243] In this optional embodiment, there is a corresponding overlap range for both the vertical overlap degree and the horizontal overlap degree, such as 10%-30%, to reduce the situation where the target is cropped or missed during the cropping process. Optionally, the number of blocks that the first image needs to be chunked into can be dynamically adjusted according to the resolution of the first image. Among them, the higher the resolution required after chunking, the smaller the chunk size.

[0244] It can be seen that the embodiment of the present invention determines the chunk overlap degree from multiple angles such as the number of blocks required for chunking the image, the size of the image, the size of the target in the image, and the step length in the horizontal and vertical directions, which can improve the accuracy and reliability of determining the chunk overlap degree, increase the probability that the target in the image is divided into the same chunk image, thereby improving the chunk accuracy and efficiency of the image, and further improving the detection efficiency and accuracy of the target.

[0245] In yet another optional implementation manner, the method may further include the following steps:

[0246] During the flight of the flying device, the embedded device performs an image acquisition operation on the current scene to obtain a video stream of the current scene;

[0247] The embedded device performs a target detection operation on the current frame image of the video stream of the current scene based on the YOLO model to obtain the detection result of the current frame image. The detection result of the current frame image includes multiple detection frames, the confidence level and category of each detection frame;

[0248] When the current frame image is the first frame image of the video stream, the embedded device assigns a corresponding identifier to each target in the current frame image according to the detection result of the current frame image, and sends the current frame image to the server. The first image includes the first frame image of the video stream;

[0249] When the current frame image is a non-first frame image of the video stream, the embedded device obtains the detection result of the previous frame image, and performs a tracking operation on the target according to the detection result of the current frame image and the detection result of the previous frame image to obtain the target tracking result of the current frame image;

[0250] Based on the object tracking result of the current frame image, the embedded device determines whether a new object appears in the current frame image. When it is determined that a new object appears in the current frame image, the embedded device performs an image acquisition operation with the target resolution to obtain a second image, and sends the second image to the server. The first image includes the second image, where the target resolution is greater than the resolution of the video stream.

[0251] In this optional embodiment, optionally, when a frame of image is acquired, the operations of this optional embodiment are performed on this frame of image. The YOLO model here and the YOLO model integrated in the server can be the same YOLO model, or a more lightweight YOLO model. However, in either case, the object network of the YOLO model here incorporates an attention module, where the attention module includes a spatial attention module and / or a channel attention module. Here, the attention module is a module determined for each channel of the feature map output by the object network of the YOLO model, and the object network of the YOLO model includes the backbone network and / or the multi-scale fusion network of the YOLO model.

[0252] In this optional embodiment, for the first frame image, it can be directly sent to the server, or the image can be re-acquired with the target resolution and then sent to the server.

[0253] In this optional embodiment, optionally, when it is determined that no new object appears in the current frame image, continue to acquire images and detect the images based on the YOLO model.

[0254] In this optional embodiment, when a new object is detected, the embedded device simultaneously sets a corresponding identifier for the new object so as to track the new object according to the corresponding identifier.

[0255] In this optional embodiment, optionally, the real-time detection frame rate used by the embedded device for image acquisition and detection is ≥ the preset frame rate, such as 30 FPS.

[0256] It can be seen that in this alternative embodiment, a YOLO model integrating an attention module with a backbone network and / or a multi-scale fusion network is integrated into an embedded device, which improves the computing power of the embedded device. Using it to perform object detection operations on the collected images can not only accurately identify small objects and edge objects in low-resolution images, but also accurately extract the structural information of objects in high-resolution images, improving the recognition efficiency and accuracy of multiple objects, as well as reducing the object missed detection rate; and first performing object detection and capture at a lower resolution by the embedded device, and only triggering a higher resolution for image acquisition and sending to the server when a new object is detected, reducing the frequent backhaul of high-resolution images and the situation where objects are repeatedly backhauled to the server, thus saving bandwidth and computing resources; and then performing collaborative processing on the images containing new objects by the server for refined recognition, improving the image data processing efficiency and balancing the requirements of object detection efficiency and accuracy, which is particularly suitable for the inspection scenario of high-resolution images.

[0257] In this alternative embodiment, as an alternative implementation, the embedded device performs a tracking operation on the object based on the detection result of the current frame image and the detection result of the previous frame image to obtain the object tracking result of the current frame image, including:

[0258] The embedded device analyzes the current information of each second object in the current frame image according to the detection result of the current frame image, and the current information of each second object includes the current position of the second object.

[0259] The embedded device analyzes the historical position of each second object in the previous frame image according to the detection result of the previous frame image, and predicts the predicted position of each second object at the next moment according to the current position and the historical position of each second object.

[0260] The embedded device performs a tracking operation on the second object according to the predicted position of each second object to obtain the object tracking result of the current frame image.

[0261] In this alternative embodiment, optionally, by analyzing the coordinates of the detection box of each second object, the position of the second object, such as the current position and the historical position, can be obtained. Further optionally, the predicted position of each second object at the next moment can be predicted by any method capable of realizing position prediction, such as a Kalman filter; and a tracking operation can be performed on each second object by any method capable of realizing object tracking, such as the Hungarian algorithm.

[0262] It can be seen that this alternative embodiment predicts the position of the target at the next moment based on the position of the currently detected target and its historical position, improving the accuracy and efficiency of target tracking. This is conducive to improving the accuracy and efficiency of discovering and distinguishing new targets, and further conducive to improving the accuracy and efficiency of identifying multiple targets.

[0263] In this alternative embodiment, as an alternative implementation, the embedded device determines whether a new target appears in the current frame image according to the target tracking result of the current frame image, including:

[0264] When the operation of assigning identifiers to the targets that appeared in the historical frame images has been performed, the embedded device determines whether the target tracking result of the current frame image is used to indicate that there are targets in the current frame image that have not been assigned identifiers. When the determination result is yes, it is determined that a new target appears in the current frame image, and the new target is the target that has not been assigned an identifier; or,

[0265] The embedded device calculates the cross value between the current position of each second target and the predicted position of the second target according to the target tracking result of the current frame image, and determines whether the cross value corresponding to each second target is greater than or equal to a preset cross value. When it is determined that it is greater than or equal to the preset cross value, it is determined that a new target appears in the current frame image, and the new target is the target that has not been assigned an identifier.

[0266] In this alternative embodiment, optionally, when it is determined that all targets in the current frame image have been assigned corresponding identifiers, and / or the cross value between the current position and the predicted position of each second target is small, it is determined that no new target appears in the current frame image.

[0267] It can be seen that this alternative embodiment determines new targets by the identifiers of the targets in the image or the cross situation between the current position and the predicted position, improving the accuracy and efficiency of new target determination. This is conducive to further improving the accuracy and efficiency of triggering high-resolution image acquisition, and further conducive to improving the efficiency and accuracy of accurately identifying new targets.

[0268] Embodiment 2

[0269] Please refer to Figure 2 , Figure 2 which is a schematic flowchart of an optimization processing method for a YOLO model applied to target detection disclosed in an embodiment of the present invention. Among them, Figure 2 The described method can be applied to any scenario that requires target detection. Among them, this scenario includes but is not limited to border patrol scenarios, power inspection scenarios, and forest fire prevention scenarios. And a corresponding flying device is set in this scenario. Optionally, the flying device is performing a corresponding flying task in the corresponding scenario. Such as Figure 2As shown, the method may include the following operations:

[0270] 201. Obtain the feature maps output by different target layers of the multi-scale fusion network of the YOLO model.

[0271] 202. For any three adjacent target layers, determine the corresponding third convolution kernel according to the resolution of the feature map of the target layer at the higher level and the resolution of the feature map of the target layer at the middle level, and based on this third convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the higher level to obtain the transfer feature map of the target layer at the higher level.

[0272] 203. Determine the corresponding fourth convolution kernel according to the resolution of the feature map of the target layer at the lower level and the resolution of the feature map of the target layer at the middle level, and based on this fourth convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the lower level to obtain the transfer feature map of the target layer at the lower level. The scales of the transfer feature map of the target layer at the lower level, the transfer feature map of the target layer at the higher level, and the feature map of the target layer at the middle level are equal.

[0273] 204. Perform a channel dimension concatenation operation on the transfer feature map of the target layer at the lower level, the transfer feature map of the target layer at the higher level, and the feature map of the target layer at the middle level to obtain a channel concatenated feature map.

[0274] 205. Based on the pre-determined fifth convolution kernel, perform a convolution operation on this channel concatenated feature map to obtain a second multi-scale fusion feature map.

[0275] It can be seen that implementing Figure 2 The described method enhances the multi-directional smooth interaction of each layer of features by adding horizontal connections in the up and down transfer paths of the FPN to directly transfer information across layers to features at the same scale, thereby improving the feature information sharing ability. And through horizontal connections and convolution operations, each layer of features contains richer context information, improving the ability of the YOLO model to capture details, especially more effective for the detection of small-sized targets, and at the same time reducing detection misses caused by target size differences, improving the detection effect for small-sized targets and complex scenes, especially suitable for complex scenes with small target areas and large quantities, such as drone scenes.

[0276] It should be noted that for other technical contents such as pruning, number of channels, channel matrix, and fine-tuning of the YOLO model, please refer to the specific descriptions of the relevant contents in Embodiment 1, which will not be elaborated in this embodiment of the present invention.

[0277] Embodiment 3

[0278] Please refer to Figure 3 , Figure 3It is a schematic structural diagram of an optimization processing device for a YOLO model applied to object detection according to an embodiment of the present invention. Among them, the device can be applied to any scenario that requires object detection. Among them, the scenario includes but is not limited to border patrol scenarios, power inspection scenarios, and forest fire prevention scenarios. And a corresponding flying device is set in the scenario. Optionally, the flying device is performing a corresponding flying task in the corresponding scenario. Such as Figure 3 As shown, the device includes:

[0279] An integration module 301, configured to integrate an attention module into the target network of the YOLO model to be optimized. The target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model. The attention module includes a channel attention module and / or a spatial attention module;

[0280] A pooling module 302, configured to perform an average pooling operation and a maximum pooling operation matching the attention module on each channel of the target feature map output from the target network based on the attention module, to obtain an average pooling result and a maximum pooling result matching the attention module;

[0281] A generation module 303, configured to generate an attention feature map of the target feature map according to the average pooling result and the maximum pooling result matching the attention module and the target feature map corresponding to the target network; wherein, the attention feature map of the target feature map is used as the input feature map of the next layer network of the attention module.

[0282] It can be seen that by implementing Figure 3 The described device optimizes the YOLO model by integrating the attention module into the backbone network and / or the multi-scale fusion network of the YOLO model, and can obtain a YOLO model that emphasizes important feature regions in the image, so that when the YOLO model performs object recognition in the image, it can accurately recognize small objects and edge objects in low-resolution images, and can accurately extract the structural information of the objects in high-resolution images, thereby facilitating improving the recognition accuracy and efficiency of various-sized objects in the image, and meeting the requirements of the refined inspection task; and the optimized YOLO model not only has high accuracy, but also has lightweight characteristics, and can be flexibly deployed on embedded devices and servers to meet the object recognition needs of various scenarios.

[0283] In an embodiment of the present invention, optionally, when the attention module includes a channel attention module, the specific manner in which the pooling module 302 performs an average pooling operation and a maximum pooling operation matching the attention module on each channel of the target feature map output from the target network to obtain an average pooling result and a maximum pooling result matching the attention module includes:

[0284] For each channel of the target feature map output from the target network based on the channel attention module, perform average pooling operation and max pooling operation to obtain the channel average pooling result and channel max pooling result of each channel of the target feature map;

[0285] Among them, the specific manner in which the generation module 303 generates the attention feature map of the target feature map according to the average pooling result and max pooling result matching the attention module and the target feature map corresponding to the target network includes:

[0286] For any channel of the target feature map, determine the target weight of this channel according to the channel average pooling result and channel max pooling result of this channel;

[0287] Generate the channel attention feature map of the target feature map according to the target weight of each channel of the target feature map and the target feature map output by the target network;

[0288] Among them, the specific manner in which the generation module 303 determines the target weight of any one of the channels of the target feature map according to the channel average pooling result and channel max pooling result of this channel includes:

[0289] For any channel of the target feature map, perform an addition operation on the channel average pooling result and channel max pooling result of this channel to obtain the channel sum pooling result of this channel;

[0290] Input the channel sum pooling result of this channel into a pre-determined first fully connected layer, and perform a non-linear transformation operation on the channel sum pooling result of this channel input into the first fully connected layer based on a pre-determined first activation function to obtain the transformation result of this channel;

[0291] Input the transformation result of this channel into a pre-determined second fully connected layer, and perform a linear transformation operation on the transformation result of this channel input into the second fully connected layer based on a pre-determined second activation function to obtain the target weight of this channel.

[0292] It can be seen that implementing Figure 3 The described device can also extract important channel information by performing global average pooling and global max pooling on each channel of the feature map, and generate corresponding channel weights through shared fully connected layers and activation functions, that is, assign weights to the feature map from the channel dimension, enhance the importance of the feature map representation by assigning different weights to different channels, thereby emphasizing the channels in the input feature map that contribute significantly to the target task in the YOLO model, suppressing irrelevant or unimportant channels, and further improving the expression of important features and the recognition accuracy and reliability of the YOLO model for various sizes of targets.

[0293] In an embodiment of the present invention, optionally, when the attention module includes a spatial attention module, the pooling module 302 performs an average pooling operation and a max pooling operation that match the attention module on each channel of the target feature map output from the target network. The specific manner of obtaining the average pooling result and the max pooling result that match the attention module includes:

[0294] Based on the spatial attention module, perform a channel average pooling operation and a channel max pooling operation on the spatial positions of each channel of the target feature map output from the target network to obtain the spatial average pooling result and the spatial max pooling result of the target feature map;

[0295] Among them, the specific manner in which the generation module 303 generates the attention feature map of the target feature map according to the average pooling result and the max pooling result that match the attention module and the target feature map corresponding to the target network includes:

[0296] Perform a concatenation operation on the spatial average pooling result and the spatial max pooling result of the target feature map to obtain the spatial concatenation feature result of the target feature map;

[0297] Generate the spatial attention feature map of the target feature map according to the concatenation feature result of the target feature map and the target feature map output from the target network;

[0298] Among them, the specific manner in which the generation module 303 generates the spatial attention feature map of the target feature map according to the concatenation feature result of the target feature map and the target feature map includes:

[0299] Based on a pre-determined first convolution kernel, perform a convolution operation on the concatenation feature result of the target feature map to obtain the convolution result of the target feature map, and based on a pre-determined third activation function, perform a normalization operation on the convolution result of the target feature map to obtain the normalization result corresponding to the target feature map; and generate the spatial attention feature map of the target feature map based on the normalization result corresponding to the target feature map and the target feature map output from the target network.

[0300] It can be seen that the implementation Figure 3The described device can also perform channel - dimension average pooling and channel - dimension max pooling on all spatial positions of each channel by using global average pooling and global max pooling methods, thereby compressing the feature map in the channel dimension and generating weights for each spatial position, so that the YOLO model can pay more attention to the regions contributing to the task. This mechanism compresses the high - dimensional information of the feature map and converts it into two - dimensional features, thereby generating a spatial - level attention map to strengthen the features at spatial positions; and combined with the aforementioned channel attention weighting, the YOLO model can more accurately focus on the target region, suppress irrelevant regions and unimportant features. In tasks such as multi - scale and small - object detection, it can significantly improve the performance of the YOLO model, enhance the attention and expression of key features, ultimately improve the accuracy and robustness of the YOLO model, and further improve the recognition accuracy of various - sized objects and multi - objects, as well as improve the adaptability of various application scenarios.

[0301] In an alternative embodiment, Figure 4 is a schematic structural diagram of an optimization processing device for a YOLO model applied to object detection according to an embodiment of the present invention. As Figure 4 shown, the device may further include:

[0302] An acquisition module 304, configured to acquire feature maps output by different target layers of the multi - scale fusion network of the YOLO model;

[0303] A first determination module 305, configured to, for any two adjacent target layers, determine a second convolution kernel that matches the attribute parameter according to the attribute parameters of the feature maps of the two target layers;

[0304] A convolution module 306, configured to perform a convolution operation that matches the attribute parameter on the feature map of the target layer that matches the second convolution kernel based on the second convolution kernel, to obtain a convolution feature map corresponding to the second convolution kernel; wherein, the attribute parameters of the convolution feature map corresponding to the second convolution kernel are the same as the attribute parameters of the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel, and the attribute parameter includes the number of channels and / or spatial resolution;

[0305] A fusion module 307, configured to perform a feature fusion operation on the convolution feature map corresponding to the second convolution kernel and the feature map of the other target layer of the two target layers except the target layer that matches the second convolution kernel, to obtain a first multi - scale fusion feature map after fusion.

[0306] It can be seen that implementing Figure 4The described device can, by simultaneously utilizing the FPN method of top-down fusion and bottom-up fusion, not only transfer high-level features rich in semantic information to low-level features with high spatial resolution, but also gradually transfer low-level features rich in detail information to high-level features rich in semantic information, enabling closer interaction of feature information at different scales, achieving full fusion of multi-scale features, significantly enhancing the multi-scale detection performance of the YOLO model, further improving the recognition accuracy of the YOLO model in small-object and multi-object scenarios, and meeting the refined requirements for object detection.

[0307] In yet another alternative embodiment, as Figure 4 shown, the device may further include:

[0308] An acquisition module 304, configured to acquire feature maps output by different target layers of the multi-scale fusion network of the YOLO model;

[0309] A first determination module 305, configured to, for any adjacent three target layers, determine a corresponding third convolutional kernel according to the resolution of the feature map of the target layer at the high level and the resolution of the feature map of the target layer at the middle level;

[0310] A convolution module 306, configured to perform a resolution convolution operation on the feature map of the target layer at the high level based on the third convolutional kernel to obtain a transfer feature map of the target layer at the high level;

[0311] The first determination module 305 is further configured to determine a corresponding fourth convolutional kernel according to the resolution of the feature map of the target layer at the low level and the resolution of the feature map of the target layer at the middle level;

[0312] The convolution module 306 is further configured to perform a resolution convolution operation on the feature map of the target layer at the low level based on the fourth convolutional kernel to obtain a transfer feature map of the target layer at the low level, and the scales of the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level are equal;

[0313] A splicing module 308, configured to perform a channel dimension splicing operation on the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level to obtain a channel-spliced feature map;

[0314] The convolution module 306 is further configured to perform a convolution operation on the channel-spliced feature map based on a pre-determined fifth convolutional kernel to obtain a second multi-scale fusion feature map.

[0315] It can be seen that implementing Figure 4The described device can also enhance the ability to share feature information by directly transmitting information across layers to features of the same scale through adding lateral connections in the up and down transmission paths of the FPN, strengthening the multi-directional smooth interaction of features in each layer, and enabling each layer of features to contain richer context information through lateral connections and convolution operations, thereby improving the ability of the YOLO model to capture details, being particularly effective for detecting small-sized targets, reducing detection omissions caused by differences in target sizes, improving the detection effect for small-sized targets and complex scenes, and being particularly suitable for complex scenes with small target areas and a large number of targets, such as drone scenes.

[0316] In yet another alternative embodiment, as Figure 4 shown, the device may further include:

[0317] A second determination module 309, configured to determine multiple performance parameters of each filter among multiple filters of each first network layer of the YOLO model and scene parameters of the application scenario of the YOLO model;

[0318] An allocation module 310, configured to, for any filter, allocate corresponding performance weights to each performance parameter of the filter according to the scene parameters of the application scenario;

[0319] A calculation module 311, configured to calculate the importance of the filter according to each performance parameter and the performance weight of the performance parameter;

[0320] A deletion module 312, configured to delete filters with importance less than or equal to a preset importance from the YOLO model according to the importance of all filters, to obtain a pruned YOLO model.

[0321] In this alternative embodiment, optionally, after the generation module 303 generates the attention feature map of the target feature map based on the average pooling result and the max pooling result matching the attention module and the target feature map corresponding to the target network, or after the fusion module 307 performs a feature fusion operation on the convolution feature map corresponding to the second convolution kernel and the feature map of another target layer of the two target layers except the target layer matching the second convolution kernel to obtain a fused first multi-scale fusion feature map, or after the convolution module 306 performs a convolution operation on the channel concatenated feature map based on a pre-determined fifth convolution kernel to obtain a second multi-scale fusion feature map, the second determination module 309 may be triggered to determine multiple performance parameters of each filter among multiple filters of each first network layer of the YOLO model and scene parameters of the application scenario of the YOLO model.

[0322] It can be seen that implementing Figure 4The described device can also analyze the importance of each filter by analyzing the loss change parameters of the YOLO model after cropping the corresponding filter, the L1 norm of the corresponding filter, the scaling factor of the batch normalization of the corresponding filter, the corresponding performance requirements, and the sensitivity analysis combined with the output of the YOLO model, improving the accuracy and reliability of determining the importance of each filter, screening and removing the filters (channels) that have less impact on the final output of the YOLO model, thereby improving the pruning accuracy of the YOLO model, reducing the adverse impact on the performance of the YOLO model during the pruning process, significantly reducing the number of parameters and computational amount of the YOLO model, reducing the memory occupancy and inference latency, and improving the inference speed of the YOLO model while maintaining the prediction accuracy of the YOLO model, which is particularly suitable for low-power computing environments such as embedded devices and edge computing scenarios.

[0323] In this alternative embodiment, the multiple performance parameters of each filter include the loss change parameter of the YOLO model after cropping the filter, the L1 norm of the weight of the filter, and the scaling factor of the batch normalization of the filter;

[0324] Among them, the specific manner in which the second determination module 309 determines the loss change parameter corresponding to each filter among the multiple filters of each first network layer in the YOLO model includes:

[0325] Obtain the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter;

[0326] According to the scene parameters of the application scenario, set corresponding loss change weights for the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter respectively;

[0327] Calculate the loss change parameter of the YOLO model after cropping the filter according to the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter and the corresponding loss change weights.

[0328] It can be seen that implementing Figure 4 The described device can also analyze the classification loss change parameter, bounding box regression loss change parameter, and / or confidence loss change parameter of the filter and their corresponding loss change weights, which can improve the accuracy of determining the loss change parameter of the filter, thereby facilitating further improving the accuracy and reliability of determining the importance of the filter, and further lightweighting the YOLO model while ensuring the YOLO model, which is beneficial for more flexible deployment in lightweight embedded devices.

[0329] In yet another alternative embodiment, such as Figure 4As shown, the second determination module 309 is further configured to determine, from all the first network layers in the pruned YOLO model, all the second network layers that have performed filter pruning;

[0330] The second determination module 309 is further configured to, for any second network layer, determine the number of channels and the channel matrix of the second network layer;

[0331] And as Figure 4 shown, the apparatus may further include:

[0332] An update module 313, configured to update the number of channels and the channel matrix of the downstream network layer of the second network layer according to the number of channels and the channel matrix of the second network layer.

[0333] It can be seen that after the apparatus described in the implementation Figure 4 is capable of pruning the YOLO model, it can further adjust the number of input channels and the channel matrix of the downstream network to ensure the consistency of the structure of the YOLO model, thereby ensuring the usage accuracy of the model.

[0334] Embodiment 4

[0335] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of another optimization processing apparatus for a YOLO model applied to object detection disclosed in an embodiment of the present invention. The apparatus can be applied to any scenario that requires object detection. Among them, the scenario includes but is not limited to border patrol scenarios, power inspection scenarios, and forest fire prevention scenarios. And a corresponding flying device is set in the scenario. Optionally, the flying device is performing a corresponding flying task in the corresponding scenario. As Figure 5 shown, the apparatus may include:

[0336] A memory 401 storing executable program code;

[0337] A processor 402 coupled to the memory 401;

[0338] Further, it may further include an input interface 403 and an output interface 404 coupled to the processor 402;

[0339] Among them, the processor 402 calls the executable program code stored in the memory 401 to execute some or all of the steps in the optimization processing method for a YOLO model applied to object detection disclosed in Embodiment 1 or Embodiment 2 of the present invention.

[0340] Embodiment 5

[0341] An embodiment of the present invention discloses a computer storage medium. The computer storage medium stores computer instructions, which, when called, are used to execute some or all of the steps in the optimization processing method of the YOLO model applied to target detection disclosed in Embodiment 1 or Embodiment 2 of the present invention.

[0342] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0343] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc memories, a magnetic disk memory, a tape memory, or any other computer-readable medium capable of carrying or storing data.

[0344] Finally, it should be noted that: The optimization processing method and device of the YOLO model applied to object detection disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than limiting them; Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: They can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimization method for the YOLO model applied to object detection, characterized in that The method includes: Integrating an attention module into the target network of the YOLO model to be optimized, where the target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model, and the attention module includes a channel attention module and / or a spatial attention module; Performing average pooling operation and max pooling operation matching the attention module on each channel of the target feature map output from the target network based on the attention module, to obtain an average pooling result and a max pooling result matching the attention module; Generating an attention feature map of the target feature map according to the average pooling result and max pooling result matching the attention module and the target feature map corresponding to the target network; Wherein, the attention feature map of the target feature map is used as the input feature map of the next layer network of the attention module.

2. The optimization method of the YOLO model applied to object detection according to claim 1, characterized in that When the attention module includes the channel attention module, the performing average pooling operation and max pooling operation matching the attention module on each channel of the target feature map output from the target network based on the attention module, to obtain an average pooling result and a max pooling result matching the attention module, includes: Performing average pooling operation and max pooling operation on each channel of the target feature map output from the target network based on the channel attention module, to obtain a channel average pooling result and a channel max pooling result of each channel of the target feature map; Wherein, the generating an attention feature map of the target feature map according to the average pooling result and max pooling result matching the attention module and the target feature map corresponding to the target network, includes: For any channel of the target feature map, determining the target weight of this channel according to the channel average pooling result and channel max pooling result of this channel; Generating a channel attention feature map of the target feature map according to the target weights of each channel of the target feature map and the target feature map output from the target network; Wherein, the determining the target weight of any channel of the target feature map according to the channel average pooling result and channel max pooling result of this channel, includes: For any channel of the target feature map, performing an addition operation on the channel average pooling result and channel max pooling result of this channel, to obtain a channel sum pooling result of this channel; Inputting the channel sum pooling result of this channel into a pre-determined first fully-connected layer, and performing a non-linear transformation operation on the channel sum pooling result of this channel input to the first fully-connected layer based on a pre-determined first activation function, to obtain a transformation result of this channel; Inputting the transformation result of this channel into a pre-determined second fully-connected layer, and performing a linear transformation operation on the transformation result of this channel input to the second fully-connected layer based on a pre-determined second activation function, to obtain the target weight of this channel.

3. The optimization method of the YOLO model applied to object detection according to claim 1, characterized in that When the attention module includes the spatial attention module, performing, for each channel of the target feature map output from the target network, an average pooling operation and a max pooling operation matching the attention module to obtain an average pooling result and a max pooling result matching the attention module, includes: Based on the spatial attention module, performing a channel average pooling operation and a channel max pooling operation on the spatial positions of each channel of the target feature map output from the target network to obtain a spatial average pooling result and a spatial max pooling result of the target feature map; Wherein, generating the attention feature map of the target feature map according to the average pooling result and the max pooling result matching the attention module and the target feature map corresponding to the target network, includes: Performing a splicing operation on the spatial average pooling result and the spatial max pooling result of the target feature map to obtain a spatial splicing feature result of the target feature map; Generating the spatial attention feature map of the target feature map according to the splicing feature result of the target feature map and the target feature map output from the target network; Wherein, generating the spatial attention feature map of the target feature map according to the splicing feature result of the target feature map and the target feature map, includes: Based on a pre-determined first convolution kernel, performing a convolution operation on the splicing feature result of the target feature map to obtain a convolution result of the target feature map, and based on a pre-determined third activation function, performing a normalization operation on the convolution result of the target feature map to obtain a corresponding normalization result of the target feature map; and generating the spatial attention feature map of the target feature map according to the corresponding normalization result of the target feature map and the target feature map output from the target network.

4. The optimization method for the YOLO model applied to object detection according to any one of claims 1-3, characterized in that, The method further includes: Obtaining the feature maps output by different target layers of the multi-scale fusion network of the YOLO model; For any two adjacent target layers, determining a second convolution kernel matching the attribute parameter according to the attribute parameters of the feature maps of the two target layers, and based on the second convolution kernel, performing a convolution operation matching the attribute parameter on the feature map of the target layer matching the second convolution kernel to obtain a convolution feature map corresponding to the second convolution kernel; wherein, the attribute parameters of the convolution feature map corresponding to the second convolution kernel are the same as the attribute parameters of the feature map of the other target layer of the two target layers except the target layer matching the second convolution kernel, and the attribute parameters include the number of channels and / or the spatial resolution; Performing a feature fusion operation on the convolution feature map corresponding to the second convolution kernel and the feature map of the other target layer of the two target layers except the target layer matching the second convolution kernel to obtain a first multi-scale fusion feature map after fusion.

5. The optimization method for the YOLO model applied to object detection according to any one of claims 1-3, characterized in that, The method further includes: Obtaining the feature maps output by different target layers of the multi-scale fusion network of the YOLO model; For any three adjacent target layers, according to the resolution of the feature map of the target layer at the high level and the resolution of the feature map of the target layer at the middle level, determine the corresponding third convolution kernel, and based on this third convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the high level to obtain the transfer feature map of the target layer at the high level; according to the resolution of the feature map of the target layer at the low level and the resolution of the feature map of the target layer at the middle level, determine the corresponding fourth convolution kernel, and based on this fourth convolution kernel, perform a resolution convolution operation on the feature map of the target layer at the low level to obtain the transfer feature map of the target layer at the low level, and the scales among the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level are equal; Perform a channel dimension concatenation operation on the transfer feature map of the target layer at the low level, the transfer feature map of the target layer at the high level, and the feature map of the target layer at the middle level to obtain a channel concatenated feature map, and based on the pre-determined fifth convolution kernel, perform a convolution operation on this channel concatenated feature map to obtain a second multi-scale fusion feature map.

6. The optimization method for the YOLO model applied to object detection according to any one of claims 1-3, characterized in that, The method further includes: Determine multiple performance parameters of each filter among the multiple filters of each first network layer of the YOLO model and the scene parameters of the application scenario of the YOLO model; For any filter, according to the scene parameters of the application scenario, assign corresponding performance weights to each performance parameter of the filter, and calculate the importance of the filter according to each performance parameter and the performance weight of this performance parameter; According to the importance of all the filters, delete the filters with importance less than or equal to the preset importance from the YOLO model to obtain the pruned YOLO model.

7. The optimization method for the YOLO model applied to object detection according to claim 6, characterized in that, The multiple performance parameters of each filter include the loss change parameter of the YOLO model after pruning this filter, the L1 norm of the weight of this filter, and the scaling coefficient of the batch normalization of this filter; Among them, the determining the loss change parameter corresponding to each filter among the multiple filters of each first network layer of the YOLO model includes: Obtain the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of this filter; According to the scene parameters of the application scenario, respectively set corresponding loss change weights for the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of this filter; Calculate the loss change parameter of the YOLO model after pruning this filter according to the classification loss change parameter, the bounding box regression loss change parameter, and / or the confidence loss change parameter of this filter and the corresponding loss change weights.

8. The optimization method of the YOLO model applied to object detection according to claim 6 or 7, characterized in that The method further includes: Among the pruned YOLO model, determine all the second network layers that have performed filter pruning from all the first network layers; For any of the second network layers, determine the number of channels and the channel matrix of the second network layer, and update the number of channels and the channel matrix of the downstream network layer of the second network layer according to the number of channels and the channel matrix of the second network layer.

9. An optimization processing device for a YOLO model applied to object detection, characterized in that, The device includes: An incorporation module, configured to incorporate an attention module into a target network of a YOLO model to be optimized, where the target network of the YOLO model includes the backbone network of the YOLO model and / or the multi-scale fusion network of the YOLO model, and the attention module includes a channel attention module and / or a spatial attention module; A pooling module, configured to perform an average pooling operation and a max pooling operation matching the attention module on each channel of a target feature map output from the target network based on the attention module, to obtain an average pooling result and a max pooling result matching the attention module; A generation module, configured to generate an attention feature map of the target feature map according to the average pooling result and the max pooling result matching the attention module and the target feature map corresponding to the target network; wherein the attention feature map of the target feature map is used as an input feature map of the next network layer of the attention module.

10. An optimization processing device for a YOLO model applied to object detection, characterized in that, The device includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the optimization processing method of the YOLO model for target detection according to any one of claims 1-8.

Images