A rotating target detection method, device, equipment and readable storage medium
Through feature extraction and feature pyramid network generation, and combining the convolutional layer output heat map and prediction map, the problem of low complexity and accuracy of the rotation object detection method is solved, and more accurate rotation object positioning and direction description are achieved.
Patent Information
- Application Number
- CN202210344819.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-03-31
AI Technical Summary
The existing rotary object detection methods are complex and highly professional, and it is difficult to cope with the detection challenges of small and dense objects, dense objects and directions in remote sensing images, and the detection accuracy is low.
Using a rotational object detection method based on feature extraction backbone network and feature pyramid network, a fusion feature map is generated through feature extraction, aggregation and upsampling, and combining the convolutional layer to output vertex heat map, coronal center point heat map, dimension prediction map and direction prediction Figure, calculates the corner coordinates of the rotation target, reduces the dependence on hyperparameters and improves detection accuracy.
It effectively reduces the difficulty and professional dependence of rotating target detection, improves detection accuracy, improves the accuracy of center point positioning through coronal thermal map, avoids errors in direct prediction angles, and can more accurately describe the direction of the rotating object.
Smart Images

Figure CN114692764B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a rotating target detection method, apparatus, device, and readable storage medium. Background Art
[0002] Rotated object detection involves determining the position of any tilted object in an image, representing it as the four corner points of the region enclosing the object. Over the past few decades, numerous excellent algorithms have emerged in the field of rotated object detection, and a growing number of researchers have devoted themselves to the research of rotated object detection algorithms, resulting in increasingly improved performance and widespread application in real-world applications. However, due to the constant variability of the rotating object's state and the environment in which it resides, developing a universal and robust rotated object detection algorithm remains a challenging problem.
[0003] Because remote sensing images are captured from a bird's-eye view, the spatial scenes they contain are larger, more complex, and more diverse and numerous. Therefore, the main difficulties in remote sensing target detection include small targets, densely packed objects, and arbitrary orientations. In related technologies, both two-stage and single-stage detection methods are mostly based on anchor frames, which introduce a large number of hyperparameters into the network. However, determining these hyperparameters requires considerable expertise and continuous tuning, and it is difficult to cope with the complex target and environmental changes that may occur during tracking. Detection methods without anchor frames directly predict angles, which inevitably lead to prediction bias and have a certain impact on the accuracy of the algorithm. Summary of the Invention
[0004] The present application provides a rotating target detection method, apparatus, device and readable storage medium to solve the problems existing in the related art such as complex detection methods, strong reliance on professional expertise and low detection accuracy.
[0005] In a first aspect, a rotating target detection method is provided, comprising the following steps:
[0006] Based on the feature extraction backbone network, feature extraction is performed on the positioning image z to obtain multiple first feature maps with different depths. The image z to be located contains multiple objects to be located;
[0007] Based on the feature pyramid network, the multiple first feature maps Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z);
[0008] The plurality of second feature maps ρ are respectively i(z) upsampling to obtain multiple third feature maps of the same size, and adding the multiple third feature maps to obtain a fused feature map P(z);
[0009] Input the fused feature map P(z) into the convolution layer to obtain the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map;
[0010] The coordinates of the corner points of each object to be located in the image to be located are calculated according to the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map.
[0011] In some embodiments, the feature pyramid network is used to process the plurality of first feature maps. Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z), including:
[0012] Based on the feature pyramid network, the multiple first feature maps are respectively Perform convolution operation to obtain multiple convolution feature maps
[0013] According to the following formula, the feature maps after multiple convolutions are respectively Perform feature addition processing to obtain multiple second feature maps ρ with different receptive fields i (z);
[0014]
[0015] In some embodiments, the step of calculating the coordinates of corner points of each object to be located in the image to be located based on the vertex heat map, the coronal center heat map, the size prediction map, the bias prediction map, and the direction prediction map includes:
[0016] Obtain multiple first confidence peak points from the vertex heat map, and take each first confidence peak point as a predicted vertex to obtain multiple predicted vertices;
[0017] Obtaining a second confidence peak point of the first object to be located from the coronal center point heat map, and using the second confidence peak point as the predicted center point of the first object to be located;
[0018] Obtain the size prediction value corresponding to the prediction center point from the size prediction map, obtain the direction prediction value corresponding to the prediction center point from the direction prediction map, and obtain the first floating-point bias value corresponding to each prediction vertex and the second floating-point bias value corresponding to the prediction center point from the bias prediction map;
[0019] The corner coordinates of the first object to be located are calculated based on the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating-point bias values and the second floating-point bias values.
[0020] In some embodiments, calculating the coordinates of the corner points of the first object to be located based on the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating-point bias values and the second floating-point bias values includes:
[0021] Map the direction prediction value to the angle prediction value;
[0022] Calculate the coordinates of a first corner point of the first object to be positioned in a rectangular coordinate system with (0, 0) as the origin based on the size prediction value;
[0023] Calculate the true center point of the first object to be located in the image to be located z according to the predicted center point and the second floating point bias value;
[0024] Calculating the coordinates of a second corner point of the first object to be located in the image coordinate system based on the angle prediction value, the coordinates of the first corner point and the true center point;
[0025] Detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value;
[0026] If no actual vertex exists, the second corner point coordinates are used as the corner point coordinates of the first object to be located.
[0027] In some embodiments, after the step of detecting whether there is an actual vertex based on the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value, the method further includes:
[0028] If there is an actual vertex, the angle prediction value is updated according to the actual vertex and the true center point to obtain a new angle prediction value;
[0029] Calculating the coordinates of a third corner point of the first object to be located in the image coordinate system based on the new angle prediction value, the coordinates of the first corner point and the true center point;
[0030] When it is detected that the third corner point coordinates are not equal to the second corner point coordinates, the third corner point coordinates are used as the corner point coordinates of the first object to be located.
[0031] In some embodiments, detecting whether an actual vertex exists based on the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value includes:
[0032] Calculate the real vertex corresponding to each predicted vertex in the image to be positioned z based on the multiple predicted vertices and the multiple first floating-point bias values;
[0033] Determine whether there is a real vertex of the same category as the first object to be located within a direction threshold range in the vertex heat map, where the direction threshold range is determined based on the direction prediction value;
[0034] If there is no real vertex of the same category as the first object to be located, then there is no actual vertex;
[0035] If there is a real vertex of the same category as the first object to be located, taking the real vertex of the same category as the first object to be located as the actual vertex;
[0036] If there are multiple real vertices of the same category as the first object to be located, one real vertex is screened out from the multiple real vertices of the same category as the first object to be located based on the predicted center point and the predicted size value as the actual vertex.
[0037] In some embodiments, the convolution kernel of the convolution layer is 3×3, and the number of layers of the convolution layer is 4.
[0038] In a second aspect, a rotating target detection device is provided, comprising:
[0039] The feature extraction unit is used to extract features from the positioning image z based on the feature extraction backbone network to obtain multiple first feature maps with different depths. The image z to be located contains multiple objects to be located;
[0040] A feature aggregation unit is used to aggregate the plurality of first feature maps based on a feature pyramid network. Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z);
[0041] A feature map processing unit, which is used to process the plurality of second feature maps p respectively i (z) upsampling to obtain multiple third feature maps of the same size, and adding the multiple third feature maps to obtain a fused feature map P(z);
[0042] A convolution processing unit, which is used to input the fused feature map P(z) into a convolution layer to obtain a vertex heat map, a corona center heat map, a size prediction map, a bias prediction map, and a direction prediction map;
[0043] A coordinate calculation unit is used to calculate the corner point coordinates of each object to be located in the image to be located based on the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map.
[0044] In a third aspect, a rotating target detection device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned rotating target detection method.
[0045] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned rotating target detection method is implemented.
[0046] The beneficial effects brought about by the technical solution provided by this application include: effectively reducing the difficulty and professional dependence of rotating target detection, and effectively improving detection accuracy.
[0047] The present application provides a rotating target detection method, apparatus, device and readable storage medium, comprising: extracting features from an image to be located based on a feature extraction backbone network to obtain multiple first feature maps of different depths, wherein the image to be located contains multiple objects to be located; aggregating features of the multiple first feature maps based on a feature pyramid network to obtain multiple second feature maps with different receptive fields; upsampling the multiple second feature maps respectively to obtain multiple third feature maps of the same size, and adding the multiple third feature maps to obtain a fused feature map; inputting the fused feature map into a convolutional layer to obtain a vertex heat map, a corona center heat map, a size prediction map, a bias prediction map and a direction prediction map; and calculating the corner coordinates of each object to be located in the image to be located based on the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map. This application uses the coronal thermal map to enable the network to learn the difference between the center point and the non-center point, without introducing a large number of hyperparameters, thereby improving the network's accuracy in locating the center point, so as to more accurately predict the center point on a slender object with strong robustness; at the same time, the vertices and center points are used to describe the rotating object, avoiding the errors caused by directly predicting the angle, and using direction prediction instead of angle prediction can more accurately describe the direction of the object. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0049] Figure 1 A flowchart of a rotating target detection method provided in an embodiment of the present application;
[0050] Figure 2A schematic diagram of the data processing flow provided in the embodiment of the present application;
[0051] Figure 3 A schematic structural diagram of a rotating target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0053] The embodiments of the present application provide a rotating target detection method, apparatus, device and readable storage medium, which can solve the problems of complex detection methods, strong dependence on professional expertise and low detection accuracy existing in related technologies.
[0054] See also Figure 1 and Figure 2 As shown, the embodiment of the present application provides a rotating target detection method, comprising the following steps:
[0055] Step S10: Extract features from the image to be positioned z based on the feature extraction backbone network to obtain multiple first feature maps with different depths The image z to be located contains multiple objects to be located;
[0056] For example, in the embodiment of the present application, a residual network structure can be selected as a feature extraction backbone network to extract image features. The network can be composed of a convolution layer and a pooling layer. Therefore, the image z to be located is input into the feature extraction backbone network to obtain multiple first feature maps with different depths. Take the output of three first feature maps with different depths as an example: if the image size of the image to be located z is 800×800×3, then the output images are of sizes 512×100×100 respectively. Image size is 1024×50×50 and an image size of 2048×25×25
[0057] Step S20: Based on the feature pyramid network, the first feature maps Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z);
[0058] Furthermore, the feature pyramid network is used to process the first feature maps. Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z), including:
[0059] Based on the feature pyramid network, the multiple first feature maps are respectively Perform convolution operation to obtain multiple convolution feature maps
[0060] According to the following formula, the feature maps after multiple convolutions are respectively Perform feature addition processing to obtain multiple second feature maps ρ with different receptive fields i (z);
[0061]
[0062] For example, in the embodiment of the present application, according to the image features of different depths in step S10, feature pyramid network is used to perform feature aggregation to obtain more accurate feature maps under different receptive fields. Specifically, the first feature map obtained in step S10 is respectively as well as Perform convolution operation to obtain the convolved feature map as well as for example The image size is 256×25×25, The image size is 256×50×50, The image size is 256×100×100; then, based on formula (1), feature addition is performed to obtain the output of the feature pyramid. For example, the second feature map representing three different receptive fields is output (i.e., ρ1(z), ρ2(z), and ρ3(z), and the receptive fields gradually increase). The specific calculation expression of formula (1) is:
[0063]
[0064] The brackets “[]” in formula (1) represent the convolution operation.
[0065] Step S30: respectively analyze the plurality of second feature maps ρ i (z) upsampling to obtain multiple third feature maps of the same size, and adding the multiple third feature maps to obtain a fused feature map P(z);
[0066] For example, in the embodiment of the present application, the second feature map ρ under different receptive fields in step S20 is i(z) is upsampled to the same size and added point by point to obtain the fused feature map P(z); specifically, in order to aggregate the features under different receptive fields, ρ1(z), ρ2(z) and ρ3(z) are upsampled respectively by formula (2) to obtain the same size as the network output and then added to finally obtain the fused feature map P(z); if the size of the network output is 256×200×200, then the size of P(z) is 256×200×200, and the specific calculation expression of formula (2) is:
[0067] P(z)=[[[ρ1(z)]]]+[[ρ2(z)]]+[ρ3(z)] (2)
[0068] The brackets “[]” in formula (2) represent upsampling, and the number of brackets represents the number of upsampling times.
[0069] Step S40: Input the fused feature map P(z) into the convolution layer to obtain a vertex heat map, a corona center heat map, a size prediction map, a bias prediction map, and a direction prediction map; wherein the convolution kernel of the convolution layer is 3×3, and the number of layers of the convolution layer is 4.
[0070] Exemplarily, in an embodiment of the present application, after the fused feature map P(z) in step S30 passes through four layers of 3×3 convolutional layers, the output (OutputHeads) can be obtained: vertex heat map, corona center heat map, object size prediction map, coordinate bias prediction map and relative direction prediction map, and the final target rectangular bounding box can be calculated based on the above five feature maps.
[0071] Among them, the vertex heat map is a heat map used to locate the vertices of the object, the corona center point heat map is a heat map used to locate the center point of the object, the bias prediction map is a floating point deviation feature map used to predict the scaling of vertex and center point coordinates, the size prediction map is a feature map used to predict the size of the object, and the direction prediction map is a feature map used to predict the relative direction of the object. Figure 2 As shown, the network output f of this embodiment C (R) can be expressed as:
[0072] f C (R)=F top ,F cen ,F size ,F off ,F dir (3)
[0073] Where, F top Represents a vertex heat map of size cls×200×200, where cls represents the number of categories of objects to be located. For example, if the objects to be located are ships, airplanes, playgrounds, and bridges, then cls is 4; Fcen represents the heat map of the center of the solar corona with a size of cls×200×200, F size The feature map of the predicted object size is 2×200×200, 2 means there are two parameters, the long side and the short side; F off It represents the paranoid feature map of the predicted vertex and center coordinates with a size of 4×200×200. 4 means there are 4 parameters, which are the deviations of the center and vertex on the X and Y axes, respectively. dir Represents a feature map of the predicted relative orientation of an object with a size of 1×200×200, where 1 indicates the presence of one parameter, namely the relative orientation of the object.
[0074] Among them, compared with the Gaussian heat map, which only has a small area or even only one point under high aspect ratio targets, the coronal heat map can better enable the network to learn the difference between the center point and the non-center point, so as to improve the accuracy of the network's positioning of the center point. Therefore, this embodiment outputs the coronal center point heat map for the center point. The specific training implementation process of the coronal heat map is as follows: for a rotating target, half of its long side w is used as the heat map radius, and the heat map H covering the entire target can be obtained. w , and the half of its short side h is the heat map radius, and the heat map H around the center point can be obtained h However, H h Only a small area around the center point can be perceived. In this way, it is easy to locate the center point to other locations during the inference stage, so that a large amount of background area is included, which leads to classification errors and other problems. Therefore, this embodiment combines these two heat maps to obtain the heat map of the center point of the corona H cen , and its calculation formula is as follows:
[0075] Where μ is the coefficient that controls the radius of the heat map, which can be set to 0.125. p ,y p ) represents the coordinate position of all points in the training heat map, (x p ,y p )∈T, T represents the set of points inside the target tilted frame, (x c ,y c ) represents the true center point coordinates in the training heat map.
[0076] Step S50: Calculate the corner coordinates of each object to be located in the image to be located according to the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map.
[0077] Furthermore, the corner point coordinates of each object to be located in the image to be located are calculated based on the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map, and the direction prediction map, including:
[0078] Obtain multiple first confidence peak points from the vertex heat map, and take each first confidence peak point as a predicted vertex to obtain multiple predicted vertices;
[0079] Obtaining a second confidence peak point of the first object to be located from the coronal center point heat map, and using the second confidence peak point as the predicted center point of the first object to be located;
[0080] Obtain the size prediction value corresponding to the prediction center point from the size prediction map, obtain the direction prediction value corresponding to the prediction center point from the direction prediction map, and obtain the first floating-point bias value corresponding to each prediction vertex and the second floating-point bias value corresponding to the prediction center point from the bias prediction map;
[0081] The corner coordinates of the first object to be located are calculated based on the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating-point bias values and the second floating-point bias values.
[0082] Furthermore, the calculating of the corner coordinates of the first object to be located according to the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating-point bias values and the second floating-point bias values includes:
[0083] Map the direction prediction value to the angle prediction value;
[0084] Calculate the coordinates of a first corner point of the first object to be positioned in a rectangular coordinate system with (0, 0) as the origin based on the size prediction value;
[0085] Calculate the true center point of the first object to be located in the image to be located z according to the predicted center point and the second floating point bias value;
[0086] Calculating the coordinates of a second corner point of the first object to be located in the image coordinate system based on the angle prediction value, the coordinates of the first corner point and the true center point;
[0087] Detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value;
[0088] If no actual vertex exists, the second corner point coordinates are used as the corner point coordinates of the first object to be located.
[0089] Furthermore, the detecting whether an actual vertex exists according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value includes:
[0090] Calculate the real vertex corresponding to each predicted vertex in the image to be positioned z based on the multiple predicted vertices and the multiple first floating-point bias values;
[0091] Determine whether there is a real vertex of the same category as the first object to be located within a direction threshold range in the vertex heat map, where the direction threshold range is determined based on the direction prediction value;
[0092] If there is no real vertex of the same category as the first object to be located, then there is no actual vertex;
[0093] If there is a real vertex of the same category as the first object to be located, taking the real vertex of the same category as the first object to be located as the actual vertex;
[0094] If there are multiple real vertices of the same category as the first object to be located, one real vertex is screened out from the multiple real vertices of the same category as the first object to be located based on the predicted center point and the predicted size value as the actual vertex.
[0095] Furthermore, after the step of detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value, the method further includes:
[0096] If there is an actual vertex, the angle prediction value is updated according to the actual vertex and the true center point to obtain a new angle prediction value;
[0097] Calculating the coordinates of a third corner point of the first object to be located in the image coordinate system based on the new angle prediction value, the coordinates of the first corner point and the true center point;
[0098] When it is detected that the third corner point coordinates are not equal to the second corner point coordinates, the third corner point coordinates are used as the corner point coordinates of the first object to be located.
[0099] For example, in the embodiment of the present application, each confidence peak point in the vertex heat map (because the output of the heat map is similar to a mountain peak, the local maximum is found, and the low confidence points are filtered out according to the confidence threshold to obtain the local confidence peak point) is taken as a predicted vertex, that is, there are multiple predicted vertices in the vertex heat map. i represents the number of predicted vertices; similarly, each confidence peak point in the coronal center point heat map is regarded as a predicted center point, that is, there are multiple predicted center points in the coronal center point heat map. i represents the number of prediction center points, and each prediction center point represents a type of target, that is, an object to be located; therefore, the corresponding size prediction value, direction prediction value, and bias prediction value can be found in the corresponding position in the size prediction map, direction prediction map, and bias prediction map according to the position of each prediction center point.
[0100] Since the calculation principle of the corner point coordinates of each object to be located in the present application is the same, for the sake of simplicity of description, the following embodiments will be explained by taking the corner point coordinate calculation process of a certain object to be located as an example.
[0101] Get a certain prediction center point According to the point-to-point matching principle, the size prediction value (w i ,h i ), direction prediction value d i And the central point paranoid prediction value (ocx i ,ocy i ), vertex paranoid prediction value (otx i ,oty i ), or obtain the paranoid prediction value of the vertex corresponding to the predicted vertex from the paranoid prediction graph based on the predicted vertex (otx i ,oty i ).
[0102] Among them, the predicted relative direction prediction value d i The range is [0,360), but due to the periodicity of trigonometric functions, the range needs to be mapped to [-90,90]. Therefore, the angle prediction value θ can be obtained by the mapping function of the following formula (5): i :
[0103]
[0104] As shown in formula (6), the size prediction value (w i ,h i ) can calculate the first corner point coordinates of each corner point of the object to be located in the rectangular coordinate system with (0, 0) as the origin Among them, j is a positive integer, representing different corner points:
[0105]
[0106] As shown in formula (7), the predicted center point And the central point paranoid prediction value of the corresponding position (ocx i ,ocy i) are added together to obtain the true center point of the object to be located in the image to be located z
[0107]
[0108] Refer to formula (8), and then according to the true center point And the coordinates of the first corner And based on the angle prediction value θ i Perform a rotation transformation to obtain the coordinates of the second corner point of the object to be located in the image coordinate system
[0109]
[0110] Since some rotating objects do not have vertices (such as helicopters, balls, swimming pools, bridges, etc.), the coordinates of the second corner points calculated based on the predicted center point can be directly used as the final coordinates of the bounding box corner points of rotating objects without vertices. Therefore, this embodiment will further determine whether the object to be located has vertices.
[0111] Specifically, see formula (9), each predicted vertex in the vertex heat map And the paranoid prediction value of the corresponding position (otx i ,oty i ) Calculate the real vertex corresponding to each predicted vertex in the image to be located z
[0112]
[0113] Then, according to the predicted square value, i , determine whether it can be from the direction threshold range (for example, the direction threshold range is [d i -10,d i +10]) to find the vertex with the smallest error between the predicted center point and the predicted size and the same category. Specifically, determine the vertex heat map [d i -10,d i +10] whether there is a real vertex of the same category as the object to be located. If not, the object to be located does not have an actual vertex. In this case, the coordinates of the second corner point are directly As the corner coordinates of the object to be located; if a real vertex of the same category as the object to be located is found, then the real vertex is used as the actual vertex; and if multiple real vertices of the same category as the object to be located are found, it is necessary to filter out a real vertex from these multiple real vertices based on the predicted center point and the size prediction value as the actual vertex. For example, if three real vertices of the same category as the object to be located (i.e., A1, A2, and A3) are found, then A1 is first subtracted from the predicted center point to obtain the first difference A1′, and then the first difference A1′ is subtracted from half of the size prediction value (i.e., 0.5w i ) are subtracted to obtain the second difference A1″. Similarly, the second difference A2″ of A2 and the second difference A3″ of A3 are obtained. Then, the smallest value is selected from A1″, A2″ and A3″. If A1″ is the smallest, then A1 is the actual vertex of the object to be located.
[0114] When the actual vertex is found After that, refer to formula (10), we need to calculate the actual vertex and the true center point The predicted angle θ i Update to get the new angle prediction value θ′ i :
[0115]
[0116] Finally, the new angle prediction value θ′ i Substitute into formula (9) to calculate the coordinates of the third corner point of the object to be located in the image coordinate system. The coordinates of the third corner point are the final coordinates of the corner points of the frame of the object to be located.
[0117] It can be seen that the present application provides a two-point single-stage rotating target detection method. The proposed coronal thermogram based on spatial position relationship can better learn the position information of the center point in the target area under high aspect ratio targets compared to the Gaussian thermogram, so as to more accurately predict the center point on the slender object; and the present application also uses the target vertex position prediction, and the combined center point and vertex avoids the position deviation caused by the direct prediction of the target direction, making the positioning of the rotating object more accurate, so that in aerial photography scenes such as ships, vehicles, airports and buildings, the present application can still maintain a high degree of accuracy. Therefore, the present application can more accurately learn the local feature information of the target by integrating different position information of the same target into a unified model, and the detection method from local to overall can achieve the purpose of reducing the target's false detection rate and improving the target rotation frame positioning effect.
[0118] The present application also provides a rotating target detection device, comprising:
[0119] The feature extraction unit is used to extract features from the positioning image z based on the feature extraction backbone network to obtain multiple first feature maps with different depths. The image z to be located contains multiple objects to be located;
[0120] A feature aggregation unit is used to aggregate the plurality of first feature maps based on a feature pyramid network. Perform feature aggregation to obtain multiple second feature maps ρ with different receptive fields i (z);
[0121] A feature map processing unit, which is used to process the plurality of second feature maps p respectively i (z) upsampling to obtain multiple third feature maps of the same size, and adding the multiple third feature maps to obtain a fused feature map P(z);
[0122] A convolution processing unit, which is used to input the fused feature map P(z) into a convolution layer to obtain a vertex heat map, a corona center heat map, a size prediction map, a bias prediction map, and a direction prediction map;
[0123] A coordinate calculation unit is used to calculate the corner point coordinates of each object to be located in the image to be located based on the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map.
[0124] Furthermore, the feature aggregation unit is specifically used to:
[0125] Based on the feature pyramid network, the multiple first feature maps are respectively Perform convolution operation to obtain multiple convolution feature maps
[0126] According to the following formula, the feature maps after multiple convolutions are respectively Perform feature addition processing to obtain multiple second feature maps ρ with different receptive fields i (z);
[0127]
[0128] Furthermore, the coordinate calculation unit is specifically used for:
[0129] Obtain multiple first confidence peak points from the vertex heat map, and take each first confidence peak point as a predicted vertex to obtain multiple predicted vertices;
[0130] Obtaining a second confidence peak point of the first object to be located from the coronal center point heat map, and using the second confidence peak point as the predicted center point of the first object to be located;
[0131] Obtain the size prediction value corresponding to the prediction center point from the size prediction map, obtain the direction prediction value corresponding to the prediction center point from the direction prediction map, and obtain the first floating-point bias value corresponding to each prediction vertex and the second floating-point bias value corresponding to the prediction center point from the bias prediction map;
[0132] The corner coordinates of the first object to be located are calculated based on the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating-point bias values and the second floating-point bias values.
[0133] Furthermore, the coordinate calculation unit is further configured to:
[0134] Map the direction prediction value to the angle prediction value;
[0135] Calculate the coordinates of a first corner point of the first object to be positioned in a rectangular coordinate system with (0, 0) as the origin based on the size prediction value;
[0136] Calculate the true center point of the first object to be located in the image to be located z according to the predicted center point and the second floating point bias value;
[0137] Calculating the coordinates of a second corner point of the first object to be located in the image coordinate system based on the angle prediction value, the coordinates of the first corner point and the true center point;
[0138] Detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value, and the size prediction value;
[0139] If no actual vertex exists, the second corner point coordinates are used as the corner point coordinates of the first object to be located.
[0140] Furthermore, the coordinate calculation unit is further configured to:
[0141] If there is an actual vertex, the angle prediction value is updated according to the actual vertex and the true center point to obtain a new angle prediction value;
[0142] Calculating the coordinates of a third corner point of the first object to be located in the image coordinate system based on the new angle prediction value, the coordinates of the first corner point and the true center point;
[0143] When it is detected that the third corner point coordinates are not equal to the second corner point coordinates, the third corner point coordinates are used as the corner point coordinates of the first object to be located.
[0144] Furthermore, the coordinate calculation unit is further configured to:
[0145] Calculate the real vertex corresponding to each predicted vertex in the image to be positioned z based on the multiple predicted vertices and the multiple first floating-point bias values;
[0146] Determine whether there is a real vertex of the same category as the first object to be located within a direction threshold range in the vertex heat map, where the direction threshold range is determined based on the direction prediction value;
[0147] If there is no real vertex of the same category as the first object to be located, then there is no actual vertex;
[0148] If there is a real vertex of the same category as the first object to be located, taking the real vertex of the same category as the first object to be located as the actual vertex;
[0149] If there are multiple real vertices of the same category as the first object to be located, one real vertex is screened out from the multiple real vertices of the same category as the first object to be located based on the predicted center point and the predicted size value as the actual vertex.
[0150] Furthermore, the convolution kernel of the convolution layer is 3×3, and the number of layers of the convolution layer is 4.
[0151] It can be seen that the present application uses the coronal thermal map to enable the network to learn the difference between the center point and the non-center point, without introducing a large number of hyperparameters, thereby improving the accuracy of the network's positioning of the center point, so as to more accurately predict the center point on a slender object, and has strong robustness; at the same time, the vertex and center point are used to describe the rotating object, avoiding the error caused by directly predicting the angle, and using direction prediction instead of angle prediction can more accurately describe the direction of the object.
[0152] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the aforementioned rotating target detection method embodiment, and will not be repeated here.
[0153] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 3 The rotating target detection device is shown running.
[0154] An embodiment of the present application also provides a rotating target detection device, comprising: a memory, a processor, and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned rotating target detection method.
[0155] Among them, the network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.
[0156] The processor may be a CPU, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting various parts of the entire computer device using various interfaces and lines.
[0157] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.
[0158] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned rotating target detection method are implemented.
[0159] The embodiments of the present application implement all or part of the aforementioned processes, and may also be completed by instructing related hardware through a computer program. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above methods may be implemented. Among them, the computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, servers, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.
[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0162] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.
[0163] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.
Claims
1. A rotating target detection method, characterized in that: The following steps are involved: Based on the feature extraction backbone network, feature extraction is performed on the image to be positioned z to obtain multiple first feature maps with different depths. , the image z to be located contains multiple objects to be located; Based on the feature pyramid network, the multiple first feature maps Perform feature aggregation to obtain multiple second feature maps with different receptive fields ; The plurality of second feature maps are respectively Upsampling is performed to obtain multiple third feature maps of the same size, and the multiple third feature maps are added together to obtain a fused feature map ; The fused feature map Input to the convolution layer to obtain vertex heat map, corona center heat map, size prediction map, bias prediction map and direction prediction map; The coordinates of the corner points of each object to be located in the image to be located are calculated according to the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map; The step of calculating the coordinates of the corner points of each object to be located in the image to be located according to the vertex heat map, the corona center heat map, the size prediction map, the bias prediction map and the direction prediction map includes: Acquire multiple first confidence peak points from the vertex heat map, and use each first confidence peak point as a predicted vertex to obtain multiple predicted vertices; Acquire a second confidence peak point of the first object to be located from the corona center point heat map, and use the second confidence peak point as the predicted center point of the first object to be located; Obtain a size prediction value corresponding to the prediction center point from the size prediction map, obtain a direction prediction value corresponding to the prediction center point from the direction prediction map, and obtain a first floating-point bias value corresponding to each prediction vertex and a second floating-point bias value corresponding to the prediction center point from the bias prediction map; The coordinates of the corner points of the first object to be located are calculated based on a plurality of predicted vertices, a predicted center point, a size prediction value, a direction prediction value, a plurality of first floating point bias values and a second floating point bias value.
2. The rotating target detection method according to claim 1, characterized in that: The first feature graphs are processed based on a feature pyramid network. Perform feature aggregation to obtain multiple second feature maps with different receptive fields ,include: Based on the feature pyramid network, the multiple first feature maps are respectively Perform convolution operation to obtain multiple convolution feature maps ; According to the following formula, the feature maps after multiple convolutions are respectively Perform feature addition processing to obtain multiple second feature maps with different receptive fields ; 。 3. The rotating target detection method according to claim 1, characterized in that: The method of calculating the corner point coordinates of the first object to be located according to the multiple predicted vertices, the predicted center point, the size prediction value, the direction prediction value, the multiple first floating point bias values and the second floating point bias values includes: Mapping direction prediction values to angle prediction values; Calculate the coordinates of a first corner point of the first object to be located in a rectangular coordinate system with (0, 0) as the origin based on the size prediction value; Calculate the true center point of the first object to be located in the image to be located z according to the predicted center point and the second floating point bias value; Calculate the coordinates of a second corner point of the first object to be located in the image coordinate system based on the angle prediction value, the coordinates of the first corner point and the true center point; Detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value and the size prediction value; If no actual vertex exists, the second corner point coordinates are used as the corner point coordinates of the first object to be located.
4. The rotating target detection method according to claim 3, characterized in that: After the step of detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value and the size prediction value, the method further includes: If there is an actual vertex, the angle prediction value is updated according to the actual vertex and the true center point to obtain a new angle prediction value; Calculate the coordinates of a third corner point of the first object to be located in the image coordinate system based on the new angle prediction value, the coordinates of the first corner point and the true center point; When it is detected that the third corner point coordinates are not equal to the second corner point coordinates, the third corner point coordinates are used as the corner point coordinates of the first object to be located.
5. The rotating target detection method according to claim 3, characterized in that: The detecting whether there is an actual vertex according to the plurality of predicted vertices, the plurality of first floating-point bias values, the direction prediction value and the size prediction value comprises: Calculate the real vertex corresponding to each predicted vertex in the image z to be located based on the multiple predicted vertices and the multiple first floating point bias values; Determine whether there is a real vertex of the same category as the first object to be located within a direction threshold range in the vertex heat map, wherein the direction threshold range is determined based on the direction prediction value; If there is no real vertex of the same category as the first object to be located, then there is no actual vertex; If there is a real vertex of the same category as the first object to be located, taking the real vertex of the same category as the first object to be located as the actual vertex; If there are multiple real vertices of the same category as the first object to be located, one real vertex is selected from the multiple real vertices of the same category as the first object to be located based on the predicted center point and the predicted size value as the actual vertex.
6. The rotating target detection method according to claim 1, characterized in that: The convolution kernel of the convolution layer is 3×3, and the number of layers of the convolution layer is 4.
7. A rotating target detection device, characterized in that: include: The feature extraction unit is used to extract features from the image to be positioned z based on the feature extraction backbone network to obtain multiple first feature maps with different depths. , the image z to be located contains multiple objects to be located; A feature aggregation unit is used to aggregate the plurality of first feature maps based on a feature pyramid network. Perform feature aggregation to obtain multiple second feature maps with different receptive fields ; A feature map processing unit, which is used to process the plurality of second feature maps respectively Upsampling is performed to obtain multiple third feature maps of the same size, and the multiple third feature maps are added together to obtain a fused feature map ; A convolution processing unit, which is used to fusion feature map Input to the convolution layer to obtain vertex heat map, corona center heat map, size prediction map, bias prediction map and direction prediction map; A coordinate calculation unit, which is used to calculate the corner point coordinates of each object to be located in the image to be located according to the vertex heat map, the corona center point heat map, the size prediction map, the bias prediction map and the direction prediction map; Wherein, the coordinate calculation unit is specifically used for: Acquire multiple first confidence peak points from the vertex heat map, and use each first confidence peak point as a predicted vertex to obtain multiple predicted vertices; Acquire a second confidence peak point of the first object to be located from the corona center point heat map, and use the second confidence peak point as the predicted center point of the first object to be located; Obtain a size prediction value corresponding to the prediction center point from the size prediction map, obtain a direction prediction value corresponding to the prediction center point from the direction prediction map, and obtain a first floating-point bias value corresponding to each prediction vertex and a second floating-point bias value corresponding to the prediction center point from the bias prediction map; The coordinates of the corner points of the first object to be located are calculated based on a plurality of predicted vertices, a predicted center point, a size prediction value, a direction prediction value, a plurality of first floating point bias values and a second floating point bias value.
8. A rotating target detection device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the rotating target detection method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the rotating target detection method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
End-to-end license plate recognition method and device
CN112541500A
Aerial image target detection method, equipment and medium
CN114140683A
Method and device for detecting target, electronic apparatus, and storage medium
WO2021082231A1
