A target detection method based on positioning information guidance

Abstract

The present invention provides a target detection method guided by positioning information, wherein positioning information represented by an intersection-and-union ratio (IoU) is embedded in a classification task as a training label, and a hierarchical IoU function discretizes a continuous IoU to better adapt to the classification task; on the one hand, compared with a discrete IoU, learning a continuous IoU in a classification task does not bring a great performance gain, but makes the optimization process slower or even non-convergent; on the other hand, for an anchor frame that does not have a high overlap with an object, learning its IoU is meaningless; therefore, the present invention improves the true label of a classification branch, adopts an IoU indicator that is consistent with the positioning accuracy of a detection frame to guide the classification task, and also adopts a hierarchical IoU function to hierarchically process the IoU vector, thereby maintaining the unity of classification and regression tasks and effectively improving the detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision recognition technology, artificial intelligence and target detection technology, and in particular relates to a target detection method based on positioning information guidance. Background Art

[0002] In the past decade, deep learning has developed rapidly and gradually become the mainstream development direction in the field of artificial intelligence. Artificial intelligence methods represented by deep learning have brought breakthrough developments in computer vision, natural language processing and other fields. As an important field in computer technology, computer vision has always received extensive attention from academia and industry. Thanks to the revolutionary progress of deep learning, artificial intelligence has also brought new ideas and methods to the field of computer vision and achieved very good results.

[0003] Object detection is a basic task in computer vision, which aims to identify and detect objects that appear in an image. The task can be specifically defined as: given an input image, determine whether the image contains an object, and if so, give the location of the object and its category. Object detection is the basis of applications such as target tracking, object segmentation, and assisted navigation, so it has a wide range of application scenarios and great significance. However, traditional object detection algorithms have reached bottlenecks in terms of detection accuracy and running speed, and cannot meet the needs of a wider range of applications. Object detection technology is in urgent need of innovation.

[0004] The application of deep learning technology in this field has broken the constraints of traditional methods and achieved great success. A series of algorithms represented by convolutional neural networks can efficiently and automatically extract features from images for target detection tasks, achieving speed and accuracy far exceeding traditional algorithms. The convolutional neural network-based model first extracts feature maps from the image to be detected through convolution and pooling operations. Then, based on the extracted multi-scale features, the offset of the initially set prior box (also called the anchor box) relative to the target in the image to be detected is predicted, and the category of the target is predicted. When outputting the detection results, the redundant detection boxes are suppressed (also called non-maximum suppression) to obtain sparse and effective detection boxes, thereby completing the detection of targets in the image.

[0005] At present, most algorithms based on convolutional neural networks split the target detection task into two subtasks: classification and positioning, and make predictions separately without interfering with each other. In the non-maximum suppression stage, the prediction box is selected based on the level of the classification prediction result, and the detection result is output. However, the algorithm handles classification and positioning separately, which leads to a large difference between the two tasks, that is, the classification score cannot represent the positioning accuracy of the prediction box. This will make the detection results output after non-maximum suppression unreliable and reduce the detection performance. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a target detection method based on positioning information guidance, which can better adapt to classification tasks and improve detection accuracy.

[0007] A target detection method based on positioning information guidance, inputting a test image into a trained target detection model to obtain the category and positioning result of the object contained in the test image; wherein the training method of the target detection model comprises the following steps:

[0008] S1: Use the first convolution branch to extract features from the sample image to obtain a feature map, and set multiple prior rectangular anchor frames on the feature map, where the number and position of the prior rectangular anchor frames are set to meet the coverage of the entire feature map;

[0009] S2: Use the second convolution branch and the third convolution branch to perform convolution operations on the feature map respectively, and obtain the classification matrix S of the prior rectangular anchor box and the offset matrix D = B + P of the prior rectangular anchor box, where B is the position matrix of the prior rectangular anchor box and P is the offset matrix of the prior rectangular anchor box relative to itself;

[0010] S3: Construct a total loss function L, calculate and determine whether the total loss function L is less than a set value. If not, change the convolution layer parameters of the first to third convolution branches, and repeat steps S1 to S3 until the total loss function L is less than the set value; if less than, the first to third convolution branches are the final target detection model;

[0011] The total loss function L is constructed as follows:

[0012] S31: Obtain the intersection-over-union ratios between each prior rectangular anchor box and each object bounding box in the sample image respectively, and obtain an M×N-dimensional intersection-over-union ratio matrix IoU, where M is the number of prior rectangular anchor boxes and N is the number of object bounding boxes;

[0013] S32: Extract the maximum element from each row of the intersection-over-union matrix IoU to construct an M×1-dimensional intersection-over-union vector IoU max , and respectively match the position of the prior rectangular anchor box corresponding to the row where each maximum element is located with the position of the object bounding box corresponding to the column where it is located, and obtain the object position matrix G′ that each prior rectangular anchor box expects to learn. At the same time, the category number of each object in G′ forms an M×1-dimensional category vector CLS;

[0014] S33: Using hierarchical intersection-over-union function to compare the IoU vector max Perform hierarchical processing to obtain a hierarchical vector hIoU;

[0015] S34: The total loss function L of the target detection model is constructed based on the hierarchical vector hIoU, the category vector CLS, the classification matrix S and the offset matrix D as follows:

[0016] L=|S-hIoU·hIoU T ·OneHot(CLS)|+|D[pos]-G′[pos]|

[0017] Among them, T is transpose, OneHot(CLS) means converting CLS into one-hot encoding, D[pos] is the position coordinate of the prior rectangular anchor box selected as the positive sample in the offset matrix D after offset, and G′[pos] is the position coordinate of the object that the prior rectangular anchor box selected as the positive sample expects to learn.

[0018] Furthermore, the method for selecting the prior rectangular anchor box as a positive sample is:

[0019] The following hierarchical intersection-over-union function is used to compare the intersection-over-union vector IoU max Perform hierarchical processing to obtain the hierarchical vector hIoU:

[0020]

[0021] IoU max (m) is the intersection over union (IoU) vector max The mth element in hIoU(m) is the mth element in the hierarchical vector hIoU, δ is the set interval segmentation threshold, and E(·) is the floor function;

[0022] Will satisfy IoU max The prior rectangular anchor boxes with (m)>0.5 are used as positive samples for training.

[0023] Furthermore, the calculation formula of the intersection-over-union ratio between each prior rectangular anchor box and each object bounding box in the sample image is as follows:

[0024]

[0025] Among them, IoU represents the intersection-over-union ratio between any two prior rectangular anchor boxes and the object bounding box, A∩B represents the intersection of any two prior rectangular anchor boxes and the object bounding box, and A∪B represents the union of any two prior rectangular anchor boxes and the object bounding box.

[0026] Furthermore, after the image to be tested is input into the trained target detection model, the obtained category and positioning results are subjected to non-maximum suppression and redundant frame elimination, and the results after non-maximum suppression and redundant frame elimination are used as the final category and positioning results of the objects contained in the image to be tested.

[0027] Furthermore, the classification matrix S of the prior rectangular anchor box is a matrix of M×C dimensions, where C is the number of all possible categories of the object, and each element in the classification matrix S represents the probability value of each prior rectangular anchor box belonging to each category;

[0028] The category corresponding to the maximum element in each row of the classification matrix S is taken as the category to which the prior rectangular anchor box corresponding to each row belongs, and each maximum element is the category score of the category to which each prior rectangular anchor box belongs.

[0029] Beneficial effects:

[0030] 1. The present invention provides a target detection method guided by positioning information, which embeds positioning information represented by IoU into a classification task as a training label, and a hierarchical IoU function discretizes the continuous IoU to better adapt to the classification task; on the one hand, compared with discrete IoU, learning continuous IoU in the classification task does not bring a great performance gain, but makes the optimization process slower or even non-convergent; on the other hand, for an anchor frame that does not overlap with the object, learning its IoU is meaningless; therefore, the present invention improves the true label of the classification branch, adopts an IoU indicator consistent with the positioning accuracy of the detection frame to guide the classification task, and also adopts a hierarchical IoU function to hierarchically process the IoU vector, which can maintain the unity of classification and regression tasks and effectively improve the detection accuracy.

[0031] 2. The present invention provides a target detection method guided by positioning information, which expands each category number in the category vector CLS into a one-hot encoding form. Then, after hierarchical IoU weighting, OneHot (CLS) can obtain the desired discrete IoU, thereby optimizing the representation of IoU in classification tasks. Then, the present invention uses the IoU as the true label to replace the traditional one-hot encoding, thereby introducing fine-grained discrete positioning information supervision in the classification task, which can better adapt to the classification task and achieve more accurate prediction result performance evaluation.

[0032] 3. The present invention provides a target detection method guided by positioning information, which uses a hierarchical intersection-and-union processing function to directly set the intersection-and-union ratio labels of these anchor frames to 0, which not only facilitates algorithm optimization convergence, but also does not affect the final detection performance.

[0033] 4. The present invention provides a target detection method guided by positioning information, in which each element in the classification matrix S represents the probability value of each prior rectangular anchor box belonging to each category. In addition to obtaining the final category of the object contained in the image to be tested, the target detection model of the present invention can also obtain the score of the final category of the object contained in the image to be tested, which is conducive to a more accurate evaluation of the classification prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of the target detection model training method provided by the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0036] Most of the existing target detection algorithms based on convolutional neural networks divide the detection task into an independent multi-classification task and a target location positioning task, which will lead to an excessive separation between the two tasks, and it is easy to make misjudgments when performing non-maximum suppression to filter repeated detection frames. The present invention provides a target detection method guided by positioning information. On the one hand, the method improves the true label of the classification branch and uses the intersection-and-union ratio indicator consistent with the detection frame positioning accuracy to guide the classification task; on the other hand, it optimizes the representation of the intersection-and-union ratio in the classification task and uses discrete positioning information supervision to better adapt to the classification task. Finally, the purpose of maintaining the unity of classification and regression tasks and improving detection accuracy is achieved.

[0037] A target detection method based on positioning information guidance, the specific implementation steps are as follows:

[0038] Input the image to be tested into the trained target detection model to obtain the initial category and initial positioning result of the object contained in the image to be tested; perform nonmaximum suppression and redundant frame elimination on the obtained initial category and initial positioning result, and use the result after nonmaximum suppression and redundant frame elimination as the final category cls of the object contained in the image to be tested * And the positioning result D * .

[0039] Among them, Figure 1 As shown, the training method of the target detection model includes the following steps:

[0040] S1: Use the first convolution branch to extract features of the sample image to obtain a feature map, and set multiple prior rectangular anchor frames on the feature map, where the number and position settings of the prior rectangular anchor frames meet the coverage of the entire feature map.

[0041] It should be noted that, for the input image I, the present invention uses a convolution operation to extract features, and performs five downsampling operations to obtain a feature map, wherein the formula of the convolution operation is:

[0042] o(i,j)=∑ m ∑ n X(m,n)·K(im,jn)

[0043] In the formula, X(m,n) represents the value of each pixel position of the input image or feature map; K(im,jn) represents the value of the convolution kernel position at the corresponding position.

[0044] At the same time, the present invention denotes a single anchor box as b = (x, y, w, h). b is a four-dimensional vector, where (x, y) represents the position of the center point of the rectangular anchor box on the feature map, and (w, h) represents the width and height of the rectangular anchor box. All anchor boxes are represented as B, which is an M×4 matrix, where M represents the total number of anchor boxes; all objects in the image are denoted as G, which is an N×4 matrix, where N represents the total number of objects.

[0045] S2: The second convolution branch and the third convolution branch are used to perform convolution operations on the feature map respectively, and the corresponding classification matrix S of the prior rectangular anchor box and the offset matrix D = B + P of the prior rectangular anchor box are obtained, where B is the position matrix of the prior rectangular anchor box, and P is the offset matrix of the prior rectangular anchor box relative to itself.

[0046] It should be noted that the second convolution branch is used as a classification branch to obtain classification prediction results, and the output S of the second convolution branch is an M×C matrix, where C is the number of all possible categories of objects. S represents the probability value of each anchor box belonging to each category; in addition, the final category score of each anchor box can be calculated by the following formula:

[0047] s=max(S,0)

[0048] cls=arg max(S,0)

[0049] The above formula indicates that the category corresponding to the maximum element in each row of the classification matrix S is used as the category of the prior rectangular anchor box corresponding to each row, and each maximum element is the category score of the category to which each prior rectangular anchor box belongs, and it is also the probability score s of all anchor boxes, and the corresponding category index cls is obtained. It can be seen that the target detection model of the present invention not only obtains the final category cls of the object contained in the image to be tested, but also obtains the category index cls of the object contained in the image to be tested. * , we can also get the final score of the category to which the object contained in the image to be tested belongs, that is, the probability.

[0050] The third convolution branch serves as a positioning branch and is used to obtain positioning prediction results. The output D of the third convolution branch is an M×4 matrix, which represents the sum of the anchor box matrix B and the offset P. That is to say, the third convolution branch first obtains the offset of each anchor box, and then superimposes the offset on the original position coordinates of each anchor box to obtain the predicted coordinates of each anchor box.

[0051] S3: Construct a total loss function L, calculate and determine whether the total loss function L is less than the set value. If not, change the convolution layer parameters of the first to third convolution branches, and repeat steps S1 to S3 until the total loss function L is less than the set value; if less than, the first to third convolution branches are the final target detection model.

[0052] The total loss function L is constructed as follows:

[0053] S31: Obtain the intersection-over-union ratios between each prior rectangular anchor box and each object bounding box in the sample image respectively, and obtain an M×N-dimensional intersection-over-union ratio matrix IoU, where M is the number of prior rectangular anchor boxes and N is the number of object bounding boxes.

[0054] It should be noted that the intersection-over-union ratio is used to measure the overlap between two bounding boxes, and is calculated as follows:

[0055]

[0056] In the formula, A and B represent two different rectangular bounding boxes; A∩B represents the intersection of A and B; A∪B represents the union of A and B. The intersection over union ratio between all anchor boxes and the target object in the image to be detected is expressed as IoU, and the formula is as follows:

[0057] IoU=IoU(B,G)

[0058] Where IoU is an M×N dimensional matrix.

[0059] S32: Extract the maximum element from each row of the intersection-over-union matrix IoU to construct an M×1-dimensional intersection-over-union vector IoU max , and respectively match the position of the prior rectangular anchor box corresponding to the row of each maximum element with the position of the object bounding box corresponding to the column, and obtain the object position matrix G′ that each prior rectangular anchor box expects to learn. At the same time, the category number of each object in G′ constitutes an M×1-dimensional category vector CLS.

[0060] That is, the present invention finds the largest element in each row and the position index of the row where it is located. These elements constitute an M×1-dimensional intersection-over-union vector, denoted as IoU max , and then assign a target with the maximum intersection-over-union ratio to each anchor frame according to the position index and let the anchor frame be responsible for predicting this target. These targets form an M×4-dimensional matrix G′, and G′ represents the matrix of the position coordinates of the object that the anchor frame expects to learn. At the same time, the category number of each object in G′ forms an M×1-dimensional category vector, denoted as CLS.

[0061] S33: Using hierarchical intersection-over-union function to compare the IoU vector maxPerform hierarchical processing to obtain a hierarchical vector hIoU.

[0062] Among them, the hierarchical intersection-and-union ratio function is as follows:

[0063]

[0064] IoU max (m) is the intersection over union (IoU) vector max The mth element in hIoU(m) is the mth element in the hierarchical vector hIoU. δ is the set interval segmentation threshold, which is responsible for dividing the intersection into discrete intervals. Here, the value is 0.1. E(·) is the floor function. The specific formula is as follows:

[0065]

[0066] Furthermore, IoU max The prior rectangular anchor box with (m)>0.5 is used as the positive sample for training and participates in the calculation of positioning loss. Its index in all anchor boxes is denoted as pos.

[0067] S34: The total loss function L of the target detection model is constructed based on the hierarchical vector hIoU, the category vector CLS, the classification matrix S and the offset matrix D as follows:

[0068] L=|S-hIoU·hIoU T ·OneHot(CLS)|+|D[pos]-G′[pos]|

[0069] Among them, T is the transpose, D[pos] is the offset position coordinate of the prior rectangular anchor box selected as the positive sample in the offset matrix D, and G′[pos] is the position coordinate of the object that the prior rectangular anchor box selected as the positive sample expects to learn. OneHot (CLS) is an M×C-dimensional matrix, which means that CLS is converted to one-hot encoding, that is, only the position where the category number is located is 1 and the other positions are 0. After the matrix is ​​hierarchically weighted by the intersection and union ratio, the desired discrete intersection and union ratio can be obtained. This intersection and union ratio is used as the true label to replace the traditional one-hot encoding. Fine-grained positioning information supervision is introduced in the classification task, which can achieve more accurate prediction result performance evaluation.

[0070] It should be noted that the total loss function L is actually composed of two parts: classification loss function and positioning loss function, as follows:

[0071] L=L cls +L loc

[0072] L cls =|S-hIoU·hIoU TOneHot(CLS)|

[0073] L loc =|D[pos]-G′[pos]|

[0074] Among them, L cls is the classification loss function, which represents the deviation between the predicted result of the classification task and the true label; L loc It is a positioning loss function, which represents the deviation between the predicted position of the anchor box as a positive sample and the position of the object that the anchor box expects to learn.

[0075] It can be seen that the general target detection algorithm uses the cross entropy loss of unique hot encoding to calculate the gap between the prediction result and the true category for the classification task, and this indicator is not related to the positioning task, resulting in a gap between the two, affecting the detection performance. The present invention embeds the positioning information represented by the intersection and union ratio into the classification task as a training label, and the hierarchical intersection and union ratio function discretizes the continuous intersection and union ratio to better adapt to the classification task. On the one hand, compared with the discrete intersection and union ratio, learning the continuous intersection and union ratio in the classification task does not bring a great performance gain, but makes the optimization process slower or even non-convergent. On the other hand, for the anchor frame with low overlap with the object, it is meaningless to learn its intersection and union ratio, so the hierarchical intersection and union ratio processing function directly sets the intersection and union ratio label of this part of the anchor frame to 0, which is convenient for the algorithm optimization convergence and will not affect the final detection performance.

[0076] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A target detection method based on positioning information guidance, characterized in that: The image to be tested is input into the trained target detection model to obtain the category and positioning result of the object contained in the image to be tested; wherein the training method of the target detection model comprises the following steps: S1: Use the first convolution branch to extract features from the sample image to obtain a feature map, and set multiple prior rectangular anchor frames on the feature map, where the number and position of the prior rectangular anchor frames are set to meet the coverage of the entire feature map; S2: Use the second convolution branch and the third convolution branch to perform convolution operations on the feature map respectively, and obtain the classification matrix S of the prior rectangular anchor box and the offset matrix D = B + P of the prior rectangular anchor box, where B is the position matrix of the prior rectangular anchor box and P is the offset matrix of the prior rectangular anchor box relative to itself; S3: Construct a total loss function L, calculate and determine whether the total loss function L is less than a set value. If not, change the convolution layer parameters of the first to third convolution branches, and repeat steps S1 to S3 until the total loss function L is less than the set value; if less than, the first to third convolution branches are the final target detection model; The total loss function L is constructed as follows: S31: Obtain the intersection-over-union ratios between each prior rectangular anchor box and each object bounding box in the sample image respectively, and obtain an M×N-dimensional intersection-over-union ratio matrix IoU, where M is the number of prior rectangular anchor boxes and N is the number of object bounding boxes; S32: Extract the maximum element from each row of the intersection-over-union matrix IoU to construct an M×1-dimensional intersection-over-union vector IoU max , and respectively match the position of the prior rectangular anchor box corresponding to the row where each maximum element is located with the position of the object bounding box corresponding to the column where it is located, and obtain the object position matrix G′ that each prior rectangular anchor box expects to learn. At the same time, the category number of each object in G′ forms an M×1-dimensional category vector CLS; S33: Using hierarchical intersection-over-union function to compare the IoU vector max Perform hierarchical processing to obtain a hierarchical vector hIoU; S34: The total loss function L of the target detection model is constructed based on the hierarchical vector hIoU, the category vector CLS, the classification matrix S and the offset matrix D as follows: L=|S-hIoU·hIoU T ·OneHot(CLS)|+|D[pos]-G′[pos]| Where T is transpose, OneHot(CLS) means converting CLS into one-hot encoding, D[pos] is the offset position coordinate of the prior rectangular anchor box selected as the positive sample in the offset matrix D, and G′[pos] is the position coordinate of the object that the prior rectangular anchor box selected as the positive sample expects to learn; The method for selecting the prior rectangular anchor box as a positive sample is: The following hierarchical intersection-over-union function is used to compare the intersection-over-union vector IoU max Perform hierarchical processing to obtain the hierarchical vector hIoU: IoU max (m) is the intersection over union (IoU) vector max The mth element in hIoU(m) is the mth element in the hierarchical vector hIoU, δ is the set interval segmentation threshold, and E(·) is the floor function; Will satisfy IoU max The prior rectangular anchor boxes with (m)>0.5 are used as positive samples for training.

2. A target detection method based on positioning information guidance as claimed in claim 1, characterized in that: The calculation formula of the intersection-over-union ratio between each prior rectangular anchor box and each object bounding box in the sample image is as follows: Among them, IoU represents the intersection-over-union ratio between any two prior rectangular anchor boxes and the object bounding box, A∩B represents the intersection of any two prior rectangular anchor boxes and the object bounding box, and A∪B represents the union of any two prior rectangular anchor boxes and the object bounding box.

3. A target detection method based on positioning information guidance as claimed in claim 1, characterized in that: After the image to be tested is input into the trained target detection model, the obtained category and positioning results are subjected to non-maximum suppression and redundant frame elimination, and the results after non-maximum suppression and redundant frame elimination are used as the final category and positioning results of the object contained in the image to be tested.

4. The target detection method based on positioning information guidance according to claim 1, characterized in that: The classification matrix S of the prior rectangular anchor box is a matrix of M×C dimensions, where C is the number of all possible categories of objects, and each element in the classification matrix S represents the probability value of each prior rectangular anchor box belonging to each category; The category corresponding to the maximum element in each row of the classification matrix S is taken as the category to which the prior rectangular anchor box corresponding to each row belongs, and each maximum element is the category score of the category to which each prior rectangular anchor box belongs.

Images