Image target detection method, system, device and storage medium

Abstract

The present invention discloses an image target detection method, system, device and storage medium, wherein the method comprises: obtaining an image to be detected, and extracting an initial multi-scale feature map of the image to be detected using a backbone network; performing feature fusion operation on the initial multi-scale feature map to generate a fused feature map; performing feature processing on the fused feature map using a BiFPN network to obtain a feature pyramid; obtaining the prediction box information and classification information of each layer of feature maps in the feature pyramid through a preset detection network, and determining the detection target of the image to be detected based on the prediction box information and classification information. The above method can reduce the number of network layers, reduce the amount of parameter calculation, and improve the calculation efficiency by performing weighted feature fusion on the initial multi-scale feature map of the image to be detected, and at the same time obtain a feature pyramid containing richer scene semantic information. Using the feature pyramid for target detection can effectively improve the detection accuracy and optimize the situation of missed detection and false detection of small targets.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image target detection method, system, device and storage medium. Background Art

[0002] In the electric power sector, intelligent distribution rooms have become one of the mainstream development directions. Among them, target detection technology in distribution room scenarios can identify and locate targets such as staff and equipment for intelligent distribution room systems, which is an important prerequisite for ensuring the reliability, safety, economy and efficiency of intelligent power grids.

[0003] However, the components of the distribution room scene are often rich, and the targets are diverse and change in real time. Therefore, how to accurately and quickly complete target detection is the core issue of the current distribution room scene target detection technology. The network structure used by the traditional distribution room scene target detection method is relatively complex, and the number of hyperparameters to be calculated is large, resulting in a long detection process, and the accuracy of the positioning prediction results is limited, and it is easy to miss small targets in the scene. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an image target detection method, system, device and storage medium, which can effectively improve the positioning prediction quality and detection rate of distribution room targets.

[0005] In a first aspect, the present invention provides an image target detection method, comprising:

[0006] Obtain an image to be detected, and use the backbone network to extract an initial multi-scale feature map of the image to be detected;

[0007] Perform feature fusion operation on the initial multi-scale feature map to generate a fused feature map;

[0008] Using the BiFPN network to perform feature processing on the fused feature map to obtain a feature pyramid;

[0009] The prediction frame information and classification information of each layer of the feature map in the feature pyramid are obtained through a preset detection network, and the detection target of the image to be detected is determined based on the prediction frame information and the classification information.

[0010] Optionally, the feature fusion operation includes a full connection operation and a sampling operation.

[0011] Optionally, the preset detection network includes a classifier and a regressor, wherein the classifier adjusts the prediction box information and the classification information based on the center positioning reliability, and the center positioning reliability is determined by DIoU and centerness.

[0012] Optionally, the extracting the initial multi-scale feature map of the image to be detected by using the backbone network is specifically: scaling and adjusting the depth, width and resolution of the backbone network, and extracting the initial multi-scale feature map of the image to be detected by using the adjusted backbone network.

[0013] Optionally, the backbone network is specifically: a multi-layer convolutional neural network; the convolution operation in the multi-layer convolutional neural network includes spatial convolution and channel convolution.

[0014] In a second aspect, the present invention further provides an image object detection system, comprising:

[0015] A feature extraction unit, used to obtain an image to be detected and extract an initial multi-scale feature map of the image to be detected using a backbone network;

[0016] A feature fusion unit is used to perform a feature fusion operation on the initial multi-scale feature map to generate a fused feature map; and perform feature processing on the fused feature map using a BiFPN network to obtain a feature pyramid;

[0017] The target detection unit is used to obtain the prediction box information and classification information of each layer of the feature map in the feature pyramid through a preset detection network, and determine the detection target of the image to be detected based on the prediction box information and classification information.

[0018] Optionally, the feature fusion operation includes a full connection operation and a sampling operation.

[0019] Optionally, the preset detection network includes a classifier and a regressor, wherein the classifier adjusts the prediction box information and the classification information based on the center positioning reliability, and the center positioning reliability is determined by DIoU and centerness.

[0020] In a third aspect, the present invention provides a data processing device, comprising a processor, wherein the processor is coupled to a memory, the memory stores a program, and the program is executed by the processor, so that the data processing device executes the image target detection method described in the first aspect.

[0021] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image target detection method as described in the first aspect above is implemented.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] The image target detection method provided by the present invention can reduce the number of network layers, reduce the amount of parameter calculation, and improve calculation efficiency by performing weighted feature fusion on the initial multi-scale feature map at the bottom layer of the image to be detected. At the same time, a feature pyramid containing richer scene semantic information is obtained. Using the feature pyramid for target detection can effectively improve the detection accuracy and optimize the situation of missed detection and false detection of small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 is a flow chart of an image target detection method provided by an embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of the network structure of the image detection model provided by an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of a feature fusion process provided by an embodiment of the present invention;

[0028] Figure 4 It is a structural diagram of an image target detection system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0031] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0034] like Figure 1 As shown, an embodiment of the present invention provides an image target detection method, comprising the following steps.

[0035] S1: Obtain the image to be detected and use the backbone network to extract the initial multi-scale feature map of the image to be detected.

[0036] In this embodiment, in order to realize real-time target detection in the power distribution room scene, it is necessary to obtain the image to be detected of the power distribution room in real time and perform target detection accordingly.

[0037] S2: Perform a feature fusion operation on the initial multi-scale feature map to generate a fused feature map; use the BiFPN network to perform feature processing on the fused feature map to obtain a feature pyramid.

[0038] The BiFPN (Bidirectional Feature Network) network is specifically a weighted bidirectional feature network, which can be used to quickly fuse multi-scale feature maps while reducing the amount of calculation.

[0039] S3: Obtain prediction box information and classification information of each layer of feature map in the feature pyramid through a preset detection network, and determine the detection target of the image to be detected based on the prediction box information and classification information.

[0040] The equipment styles in the distribution room scene image are rich and complex, and the scales of the targets vary greatly. Direct target detection without distinguishing the scales is prone to missed detection and false detection problems. The above embodiment of the present invention can reduce the number of network layers, reduce the amount of parameter calculation, and improve computing efficiency by weighted feature fusion of the initial multi-scale feature map at the bottom layer of the real-time image to be detected in the distribution room scene. At the same time, a feature pyramid containing richer scene semantic information is obtained. Using the feature pyramid for target detection can effectively improve the detection accuracy and optimize the missed detection and false detection of small targets.

[0041] The following will introduce the execution process of the image object detection method through a specific embodiment.

[0042] In this embodiment, after obtaining the current distribution room scene to be detected image, the preset image detection model is used to perform feature extraction, feature fusion and target detection. The network structure of the image detection model is as follows: Figure 2 shown.

[0043] It should be noted that in order to balance the algorithm running speed and recognition accuracy, the size of the image to be detected can be set to 384*768. In the feature extraction stage, a multi-layer convolutional neural network can be selected as the backbone network to extract the underlying feature maps C3-C5 of the image to be detected. Specifically, the backbone network is set to a residual neural network (ResNet).

[0044] Specifically, the depth, width and resolution of the backbone network can be scaled and adjusted, and the initial multi-scale feature maps C3-C5 of the image to be detected are extracted through the adjusted backbone network.

[0045] When extracting features, deepening the depth of the network can obtain more complex features, but too deep a network will face the problem of gradient vanishing; while widening the network and increasing the resolution are conducive to capturing more subtle features, it will also increase the amount of calculation and affect the speed of the algorithm. Therefore, the traditional method is mainly to perform a single scaling of the three aspects of network depth, width and resolution. The embodiment of the present application uses a composite coefficient to achieve a unified scaling adjustment of the depth, width and resolution of the network, and uses the adjusted network to extract the initial multi-scale feature maps C3, C4 and C5 of the image to be detected. Among them, the feature maps C3-C5 are 1 / 8, 1 / 16 and 1 / 32 of the scale of the original image to be detected, respectively, which are used to increase the multi-scale detection performance of targets in the distribution room scene.

[0046] In this embodiment, the grid search method can be used to determine that the network depth is expanded by 1.2 2 times, width expanded by 1.1 2 times, the image size is enlarged by 1.15 2 times.

[0047] It can be understood that the standard convolution layer usually extracts features for each dimension of the input image features and obtains a dimension of the output feature map by element-by-element addition. In order to reduce the number of parameters and improve computational efficiency, this embodiment uses depthwise separable convolution to speed up feature extraction.

[0048] Specifically, the depth-wise separable convolution decomposes the convolution operation into two steps. The first step is spatial convolution: feature extraction is performed on each channel of the input feature map; the second step is channel convolution: based on 1×1 point-by-point convolution, new features are constructed by calculating the linear combination of the input channels.

[0049] The above-mentioned depth-wise separable convolution can greatly reduce the number of network parameters and achieve fast feature extraction in image scenes.

[0050] Furthermore, a feature fusion operation is performed on the initial multi-scale feature map to generate a fused feature map, wherein the feature fusion operation includes a full connection operation and a sampling operation.

[0051] Specifically, through the horizontally connected 1×1 fully connected layer and the nearest neighbor interpolation method for upsampling, the scale of the high-level feature map is enlarged by 2 times, and then the high-level feature map is added to the feature map of the lower layer element by element. After downsampling, feature maps with scales of 1 / 64 and 1 / 128 are obtained. Finally, five levels of scale feature maps, P3, P4, P5, P6 and P7, are obtained, and the corresponding fused feature maps are 1 / 8, 1 / 16, 1 / 32, 1 / 64 and 1 / 128 of the scale of the original feature map, respectively.

[0052] Furthermore, the fusion feature maps of the five scales P3-P7 are fused through the weighted feature fusion network BiFPN with bidirectional cross-scale connections to obtain a feature pyramid, thereby enriching the multi-scale target feature information. For a schematic diagram of the feature fusion process, please refer to Figure 3 .

[0053] Specifically, taking the features of the P6 scale as an example, the features of the next layer and the input features of the sixth layer can be used to calculate the nodes in the middle of the sixth layer, and then the input of the sixth layer, the output of the middle node and the output of the previous layer can be used to calculate the fusion features of the sixth layer, that is:

[0054]

[0055]

[0056] In the formula, They are Figure 3 Input layer features of the P6 and P7 levels; It is the intermediate layer feature of P6 level; is the output layer feature of P5 and P6 levels; Resize means upsampling or downsampling, which is used to adjust the feature scales of different layers to be consistent; Conv means convolution operation; w i 、w' i It represents the learnable weight. The value range of i is 1-3, which can correspond to the weights of features at different levels. ε is set to a value close to 0.

[0057] This embodiment uses the weighted feature fusion network to identify small targets such as workers in the image in the low-level feature map, and to identify large targets such as equipment in the high-level feature map, thereby enhancing the entire feature hierarchy using accurate low-level positioning signals, thereby shortening the information path between the bottom and top feature maps, and obtaining richer semantic information about the distribution room scene.

[0058] Furthermore, during the target detection process, the present embodiment obtains prediction box information and classification information of each layer of feature graph in the feature pyramid through a preset detection network, and determines the detection target in the current image to be detected based on the obtained prediction box information and classification information.

[0059] This embodiment improves the target detection method in the FCOS full convolution target detection model and uses the improved detection network to process the feature pyramid after feature fusion.

[0060] Specifically, the improved detection network in this embodiment includes a classifier and a regressor, which can output a target recognition prediction box and a classification prediction result.

[0061] The prediction box of target recognition can be represented by the coordinates of the center point and the distances from the center point to the four sides of the prediction box. Specifically, the coordinates of the upper left corner of the i-th prediction box are set to The coordinates of the lower right corner are Then the training objective of the regressor is:

[0062]

[0063] Among them, l * Indicates the distance between the current center pixel and the left border of the candidate box, r * Indicates the distance from the right border, t * Indicates the distance from the top border, b * It indicates the distance from the bottom border.

[0064] In this embodiment, the classifier is specifically composed of a plurality of binary classifiers, which can predict each layer of feature graphs in the feature pyramid and output corresponding classification prediction results.

[0065] In order to minimize the low-quality prediction frames with large center point offsets generated during the prediction process, this embodiment uses the center positioning confidence to filter the low-quality prediction frames to improve the prediction quality and detection rate of the distribution room scene target detection and positioning.

[0066] Specifically, the center positioning confidence is determined by DIoU and centerness, which is expressed as:

[0067] Center Localization Confidence=α×centerness+(1-α)×DIoU

[0068] In the formula, centerness is used to measure whether the predicted point is close to the center point of the prediction box, DIoU is used to judge whether the predicted point is close to the real point, reflecting the degree of overlap between the prediction box and the prediction box; α is a balance factor, which is used to balance the centerness and intersection-over-union ratio, and its value range is 0 to 1.

[0069] Specifically, centrality is expressed as:

[0070]

[0071] The Centerness layer mainly searches for the center point of the target. The closer to the center point of the target, the larger the output value, and vice versa.

[0072] Specifically, DIoU can simultaneously consider the overlapping area of ​​the prediction box boundary and the distance between the two center points, which has better robustness. Its definition can be expressed as:

[0073]

[0074]

[0075] In the formula, b represents the coordinate of the center point of the prediction box, b gt Indicates the coordinates of the center point of the true value box, ρ represents the Euclidean distance between the corresponding center points of the predicted box and the true value box, and c represents the diagonal distance between the minimum circumscribed rectangle of the predicted box and the true value box. In IoU, A represents the area of ​​the predicted box and B represents the area of ​​the true box.

[0076] This embodiment achieves effective filtering of low-quality prediction frames (such as false positive frames) by multiplying the value of the center positioning confidence with the output value of the classification category prediction of the classifier, thereby improving the accuracy of target recognition.

[0077] The above-mentioned embodiment of the present invention obtains richer distribution room scene semantic information by proposing a weighted feature fusion strategy of feature extraction and composite cross, and effectively reduces the parameter calculation amount, optimizes the phenomenon of missed detection and false detection of small targets, and improves the detection accuracy and speed of the algorithm; at the same time, it also proposes the center positioning reliability, by fusing DIoU with centerness, so that the predicted point is close to the center point of the prediction box and the center of the distribution room target at the same time, effectively enhancing the correlation between the prediction score and the actual positioning quality, improving the target positioning prediction quality, and avoiding the missed detection of small targets.

[0078] See also Figure 4In a second aspect, another embodiment of the present invention further provides an image target detection system, comprising a feature extraction unit, a feature fusion unit and a target detection unit.

[0079] The feature extraction unit 101 is used to obtain an image to be detected and extract an initial multi-scale feature map of the image to be detected using a backbone network.

[0080] The feature fusion unit 102 is used to perform a feature fusion operation on the initial multi-scale feature map to generate a fused feature map; and use the BiFPN network to perform feature processing on the fused feature map to obtain a feature pyramid.

[0081] The target detection unit 103 is used to obtain the prediction box information and classification information of each layer of the feature map in the feature pyramid through a preset detection network, and determine the detection target of the image to be detected based on the prediction box information and classification information.

[0082] The information interaction, execution process, etc. between the units in the above system are based on the same concept as the image target detection method embodiment of the present invention. For specific contents, please refer to the description in the method embodiment of the present invention and will not be repeated here.

[0083] In a third aspect, the present invention provides a data processing device, comprising a processor, wherein the processor is coupled to a memory, the memory stores a program, and the program is executed by the processor, so that the data processing device executes the image target detection method described in the first aspect.

[0084] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image target detection method as described in the first aspect above is implemented.

[0085] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-monitorable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0086] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for detecting an image target, characterized in that: include: Obtain an image to be detected, and use the backbone network to extract an initial multi-scale feature map of the image to be detected; Perform feature fusion operation on the initial multi-scale feature map to generate a fused feature map; Using the BiFPN network to perform feature processing on the fused feature map to obtain a feature pyramid; Obtaining prediction frame information and classification information of each layer of feature graph in the feature pyramid through a preset detection network, and determining a detection target of the image to be detected based on the prediction frame information and classification information; The preset detection network includes a classifier and a regressor, wherein the classifier adjusts the prediction box information and the classification information based on the center localization confidence, and the center localization confidence is determined by DIoU and centerness, and is expressed as: Center Localization Confidence = α × centerness + (1-α) × DIoU; wherein centerness is used to measure whether the predicted point is close to the center point of the prediction box, and DIoU is used to judge whether the predicted point is close to the real point, reflecting the degree of overlap between the prediction box and the prediction box; is a balance factor, which is used to balance the center degree and the intersection-over-union ratio, and its value range is 0 to 1; The method of performing feature processing on the fused feature map by using the BiFPN network to obtain a feature pyramid includes: performing fusion processing on the fused feature map of five scales P3-P7 by using the BiFPN network to obtain a feature pyramid; For the output features of the P3 level scale, the output features of the third layer are calculated using the input features of the third layer and the output features of the intermediate nodes of the fourth layer; For the output features of the P4 level scale, the output features of the intermediate nodes of the fourth layer are calculated using the input features of the fourth layer and the output features of the intermediate nodes of the fifth layer, and then the output features of the fourth layer are calculated using the input features of the fourth layer, the output features of the intermediate nodes of the fourth layer, and the output features of the third layer; For the output features of the P5 level scale, the output features of the intermediate nodes of the fifth layer are calculated using the input features of the fifth layer and the output features of the intermediate nodes of the sixth layer, and then the output features of the fifth layer are calculated using the input features of the fifth layer, the output features of the intermediate nodes of the fifth layer, and the output features of the fourth layer; For the output features of the P6 level scale, the input features of the seventh layer and the input features of the sixth layer are used to calculate the output features of the intermediate nodes of the sixth layer, and then the output features of the sixth layer are calculated through the input features of the sixth layer, the output features of the intermediate nodes of the sixth layer, and the output features of the fifth layer; For the output features of the P7 level scale, the output features of the seventh layer are calculated using the input features of the seventh layer and the output features of the sixth layer.

2. The image target detection method according to claim 1, characterized in that: The feature fusion operation includes a full connection operation and a sampling operation.

3. The image target detection method according to claim 1, characterized in that: The backbone network is used to extract the initial multi-scale feature map of the image to be detected, specifically: The depth, width and resolution of the backbone network are scaled and adjusted, and an initial multi-scale feature map of the image to be detected is extracted through the adjusted backbone network.

4. The image target detection method according to any one of claims 1 to 3, characterized in that: The backbone network is specifically: Multi-layer convolutional neural network; the convolution operation in the multi-layer convolutional neural network includes spatial convolution and channel convolution.

5. An image target detection system, characterized in that: include: A feature extraction unit, used to obtain an image to be detected and extract an initial multi-scale feature map of the image to be detected using a backbone network; A feature fusion unit is used to perform a feature fusion operation on the initial multi-scale feature map to generate a fused feature map; and perform feature processing on the fused feature map using a BiFPN network to obtain a feature pyramid; A target detection unit, used to obtain prediction frame information and classification information of each layer of feature graph in the feature pyramid through a preset detection network, and determine a detection target of the image to be detected based on the prediction frame information and classification information; The preset detection network includes a classifier and a regressor, wherein the classifier adjusts the prediction box information and the classification information based on the center localization confidence, and the center localization confidence is determined by DIoU and centerness, and is expressed as: Center Localization Confidence = α × centerness + (1-α) × DIoU; wherein centerness is used to measure whether the predicted point is close to the center point of the prediction box, and DIoU is used to judge whether the predicted point is close to the real point, reflecting the degree of overlap between the prediction box and the prediction box; is a balance factor, which is used to balance the center degree and the intersection-over-union ratio, and its value range is 0 to 1; The method of performing feature processing on the fused feature map by using the BiFPN network to obtain a feature pyramid includes: performing fusion processing on the fused feature map of five scales P3-P7 by using the BiFPN network to obtain a feature pyramid; For the output features of the P3 level scale, the output features of the third layer are calculated using the input features of the third layer and the output features of the intermediate nodes of the fourth layer; For the output features of the P4 level scale, the output features of the intermediate nodes of the fourth layer are calculated using the input features of the fourth layer and the output features of the intermediate nodes of the fifth layer, and then the output features of the fourth layer are calculated using the input features of the fourth layer, the output features of the intermediate nodes of the fourth layer, and the output features of the third layer; For the output features of the P5 level scale, the output features of the intermediate nodes of the fifth layer are calculated using the input features of the fifth layer and the output features of the intermediate nodes of the sixth layer, and then the output features of the fifth layer are calculated using the input features of the fifth layer, the output features of the intermediate nodes of the fifth layer, and the output features of the fourth layer; For the output features of the P6 level scale, the input features of the seventh layer and the input features of the sixth layer are used to calculate the output features of the intermediate nodes of the sixth layer, and then the output features of the sixth layer are calculated through the input features of the sixth layer, the output features of the intermediate nodes of the sixth layer, and the output features of the fifth layer; For the output features of the P7 level scale, the output features of the seventh layer are calculated using the input features of the seventh layer and the output features of the sixth layer.

6. The image target detection system according to claim 5, characterized in that: The feature fusion operation includes a full connection operation and a sampling operation.

7. A data processing device, characterized in that: include: A processor, wherein the processor is coupled to a memory, the memory stores a program, and the program is executed by the processor, so that the data processing device executes the image object detection method according to any one of claims 1 to 4.

8. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and the computer instructions are used to execute the image object detection method according to any one of claims 1 to 4.

Images