An aerial video stream target recognition algorithm performance improvement method

Abstract

The present application relates to a kind of aerial video stream target identification algorithm performance promotion method, belong to image processing and aerial video target identification technical field.The method of the present application includes: obtaining original video stream image and the information vector of all targets that the target identification algorithm is based on original video stream image output;Based on the position of all targets, original video stream image is adaptively segmented into multiple squares and makes the center of all targets as far as possible distribution in different square;The position of the target whether need to be corrected is judged based on the correlation degree of the square area where the center point of the target is located and adjacent square area group, and corrected when it is judged to need to be corrected;Based on the information of the same target that the target identification algorithm outputs before and after N frames, the category accuracy of the target is judged by the front and rear frame association algorithm, and the target of inaccurate category judgment is deleted.The problem that the detection ability of weak and small target is not good and the robustness is poor in the current target identification technology is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and aerial video target recognition technology, and particularly relates to a method for improving performance of an aerial video stream target recognition algorithm. BACKGROUND

[0002] Target recognition is to find all objects of interest in an image, and to determine the category and position of the objects. The current mainstream technology trend is target recognition based on deep learning algorithms. The deep learning algorithms mainly focus on semantic understanding of target images, extract edge features of the images through convolution integral networks, gradually extract deep target features with deepening of the network, and detect target positions through understanding of image semantics.

[0003] The difficulties of aerial target recognition algorithms mainly include:

[0004] Due to the large number of weak (low signal-to-noise ratio) and small (occupying a small full frame) targets in the aerial scene, and the less information contained therein, the current domestic and foreign target recognition technologies generally have low detection accuracy for weak and small targets, or have harsh usage conditions, and do not have universality. Some algorithms are only effective for a certain specific scene, and some algorithms are only effective for signals of a certain characteristic band. The algorithms do not achieve plug-and-play, and have high use cost. Many algorithms contain steps of modifying a target recognition model, and once the model is modified, the user needs to provide video data again, retrain, which takes a long time.

[0005] In addition, algorithms with good recognition capability mostly have complex calculation steps, and need to additionally increase the calculation cost, and some industrial scenes (such as satellites, underwater, etc.) with tight hardware resources may not be able to provide so many calculation resources. SUMMARY

[0006] In view of the above analysis, the embodiments of the present application aim to provide a method for improving performance of an aerial video stream target recognition algorithm, to solve the problem that the current target recognition technologies generally have poor detection capability for weak and small targets and poor robustness.

[0007] The embodiments of the present application provide a method for improving performance of an aerial video stream target recognition algorithm, comprising:

[0008] obtaining an original video stream image and positions, categories and confidence levels of all targets output by the target recognition algorithm based on the original video stream image;

[0009] based on the positions of all targets, adaptively segmenting the original video stream image into multiple squares and making the centers of all targets as much as possible to be distributed in different squares;

[0010] judging whether the position of the target needs to be corrected based on the correlation degree of the target center point in the grid area and the adjacent grid area group, and correcting the position when it is judged that the position needs to be corrected;

[0011] judging the category accuracy of the target based on the front and back N frame information of the same target output by the target recognition algorithm, and deleting the target with inaccurate category judgment through a front and back frame correlation algorithm.

[0012] Further, the original video stream image is adaptively segmented into MxN grids,

[0013] M = [W / w max +k M ×δ M ],

[0014] N = [H / h max +k N ×δ N ];

[0015] wherein W and H are the pixel sizes in the width and height directions of the output image respectively; w max and h max are the maximum values of the pixels in the width and height directions of all the targets output by the target recognition algorithm respectively; δ M and δ N are the adaptive coefficients in the width and height directions respectively; k i is a loop control constant, and i is M or N, representing the width and height directions respectively.

[0016] Further, the adjacent grid area group includes an upper and lower grid area group, a left and right grid area group, and a four-corner grid area group adjacent to the grid area where the center point of the target is located; there is a partially overlapping area between each two adjacent grid areas; the upper and lower grid area group includes two grid areas C1 and C2 adjacent to the grid area A where the center point of the target is located in the upper and lower directions, the left and right grid area group includes two grid areas D1 and D2 adjacent to the grid area A in the left and right directions, and the four-corner grid area group includes four grid areas B1, B2, B3 and B4 located at the four corners of the grid area A.

[0017] Further, judging whether the position of the target needs to be corrected based on the correlation degree of the target center point in the grid area and the adjacent grid area group includes: if the position of the target detected in the next frame does not appear in the grid area A but appears in the adjacent area group with the highest correlation degree, it is considered to be correct recognition, and the target is retained; otherwise, it is judged that the position of the target needs to be corrected.

[0018] The position correction when judging the need for correction includes: obtaining a correction distance value based on the Euclidean distance of the center point position of the target in the current frame and the next frame multiplied by the similarity information of the A grid region and the grid region group where the target is detected in the next frame; correcting the next frame target center point position based on the correction distance value so that the next frame target center point position is still located on the line connecting the center points of the current frame and the next frame; if the corrected next frame target center point returns to the A grid region, the target is retained, otherwise, the confidence level of the target output by the target recognition algorithm is combined to judge whether to retain or discard the target.

[0019] Further, the center point of the target is extracted by the convolution network. A grid region A and the feature map of each grid region in each adjacent grid region group are extracted and matched and compared to obtain the adjacent grid region group with the highest correlation degree.

[0020] Further, the center point of the target is extracted by the convolution network. A grid region A and the feature map of each grid region in each adjacent grid region group are extracted and matched and compared to obtain the adjacent grid region group with the highest correlation degree.

[0021] A and the 8 adjacent grid region images are input respectively, and each grid region feature map is output by 9 3-1 structures, each 3-1 structure including 3x3 convolution and 1x1 convolution.

[0022] Further, the adjacent grid region group with the highest correlation degree is obtained, including:

[0023] The similarity information β(A, B1) of A and B1 grid regions is obtained by the following formula:

[0024]

[0025] Wherein, A i，j is the pixel value of the i-th row and j-th column in the A grid region feature map, B1 i，j is the pixel value of the i-th row and j-th column in the B1 grid region feature map.

[0026] Similarly, β(A, B2), β(A, B3), and β(A, B4) are obtained.

[0027] β(A, B) = [β(A, B1) + β(A, B2) + β(A, B3) + β(A, B4)] / 4, and β(A, C), β(A, D) are obtained similarly.

[0028] Compare β(A, B), β(A, C), and β(A, D), and take the grid region corresponding to the minimum β value as the adjacent grid region group with the highest correlation degree.

[0029] Further, based on the same target of N frames of information before and after the output of the target recognition algorithm, the category accuracy of the target is judged through the front and rear frame association algorithm, including:

[0030] The confidence of the current frame is updated according to the following formula:

[0031] P=P i + alpha i * P i-1 + alpha i * P i-2 +......+ alpha i * P1

[0032] Wherein, P i is the confidence of the current frame, P i-1 is the confidence of the last frame, P1 is the first frame confidence, alpha i is the decay coefficient of the current frame, the range is [0, 1]; if P is greater than the confidence threshold P th , it is considered that the category recognition of the target is accurate; if P is lower than the confidence threshold P th , whether the category of the target is accurate is determined by executing a multi-frame voting strategy.

[0033] Further, if P is lower than the confidence threshold P th , whether the category of the target is accurate is determined by executing a multi-frame voting strategy, including:

[0034] Taking the detection results of the last p frames, if the P of q frames is greater than the threshold P th , wherein q<p, it is considered that the detection result is accurate; if less than q frames meet the requirement, it is considered that the category detection of the target is not accurate, and the target is discarded.

[0035] Further, the modified target position and the accurate target category are fused with the obtained original video stream image to output the recognition result.

[0036] Compared with the prior art, the present application can realize at least one of the following beneficial effects:

[0037] 1. The present application provides an aviation video stream target recognition algorithm performance improvement method with low algorithm power consumption, which realizes the purpose of improving the performance of the existing target recognition algorithm.

[0038] 2. The present application provides an algorithm module that can be plugged in and used, which is highly adaptable. As long as the original video stream image and the position coordinates and category information output by the original recognition algorithm can be obtained, the position accuracy of the original target recognition algorithm can be improved, the false alarm rate can be reduced, the target category recognition accuracy of the original target recognition algorithm can be improved by combining the confidence level, and the target recognition flicker rate of the original target recognition algorithm can be reduced.

[0039] The technical solutions in the present application can be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the following description, and some advantages will become apparent from the description, or will be understood by those skilled in the art through implementation of the present application. The objects and other advantages of the present application can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of the detailed description. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application. In the drawings:

[0041] Figure 1 A flow chart of an aerial video stream target recognition algorithm performance improvement method for an embodiment of the present application;

[0042] Figure 2 A schematic diagram of MxN squares segmented by an image adaptive segmentation algorithm used in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of positions of an A square region and eight adjacent square regions in a local attention matching algorithm used in an embodiment of the present application;

[0044] Figure 4 A flow chart of a small convolutional network in a local attention matching algorithm used in an embodiment of the present application. DETAILED DESCRIPTION

[0045] Preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of the present application and serve to explain principles of the embodiments of the present application, but are not intended to limit the scope of the present application.

[0046] One specific embodiment of the present application discloses an aerial video stream target recognition algorithm performance improvement method, which aims to improve the detection accuracy of an original target recognition algorithm, as shown in the following formula: Figure 1 The method comprises the following steps:

[0047] Step S1, obtaining an original video stream image and positions, categories and confidence of all targets output by the target recognition algorithm based on the original video stream image;

[0048] The original video stream image is input to an adaptive segmentation algorithm, the position information of all targets is input to a local attention matching algorithm, the category information of all targets is input to a front-back frame association algorithm, and the confidence information of all targets is input to the local attention matching algorithm and the front-back frame association algorithm.

[0049] Step S2, based on the positions of all targets, adaptively segment the original video stream image into multiple squares and make the centers of all targets as much as possible to be distributed in different squares.

[0050] Adaptively segment the original video stream image into MxN squares as shown in Figure 2 , so that the centers of all detected targets are as much as possible to be distributed in different squares, and the targets are maximally placed in 9 squares. The specific steps include:

[0051] Calculate the segmentation number M and N in the width and height directions of the original video stream image by the formulas M=[W / w max +k M xδ M ] and N=[H / h max +k N xδ N ], respectively.

[0052] Where [] is the rounding operation, W and H are the pixel sizes in the width and height directions of the image, w max and h max are the maximum values of the pixel sizes in the width and height directions of all detected targets output by the original target recognition algorithm, δ M and δ N are the width and height adaptive coefficients when calculating M and N, smaller coefficients can make M and N as small as possible to reduce the calculation space, larger coefficients can make each target more independent in different squares to increase the accuracy, δ M and δ N are generally around 4, which can be adjusted according to needs; k i is a loop control constant, i is M or N representing the width and height directions, and is determined by the following steps:

[0053] a) Let k i =0, check whether two target centers are segmented into the same square, if not, go to c), if yes, continue b);

[0054] b) Let k i =k i +1, repeat a);

[0055] c) Check whether the target width and height pixel sizes w and h exceed 3W / M and 3H / N, respectively, if not, complete the determination of k i , if yes, continue d);

[0056] d) k i =k i -1, repeat c).

[0057] Step S3, judging whether the position of the target needs to be corrected based on the degree of association between the grid area where the center point of the target is located and the group of adjacent grid areas, and performing position correction when it is judged that correction is needed.

[0058] The local attention matching algorithm is used to match the grid area with a higher degree of association according to the feature information of the grid area where the center point of the target is located, so as to improve the stability of the position of the detected target in the next frame. Specifically, steps S31-S33 are included.

[0059] Step S31, determining the grid area A where the center point of the target is located and the group of adjacent grid areas thereof.

[0060] On the basis of dividing the acquired original video stream image into MxN grid areas in step S2, each grid area is expanded to have a partial overlap between every two adjacent grid areas, so as to stabilize the detection result and avoid the flickering phenomenon of the detection result falling on the boundary between two grid areas. The grid area A includes the grid where the center point of the target is located and the area expanded therefrom in all directions. The group of adjacent grid areas includes the group of upper and lower grid areas, the group of left and right grid areas, and the group of four corner grid areas adjacent to the grid area A; the group of upper and lower grid areas includes two grid areas C1 and C2 adjacent to A in the upper and lower directions, the group of left and right grid areas includes two grid areas D1 and D2 adjacent to A in the left and right directions, and the group of four corner grid areas includes four grid areas B1, B2, B3 and B4 located at the four corners of A, as shown in Figure 3 .

[0061] Step S32, using the local attention matching algorithm to match the group of adjacent grid areas with the highest degree of association according to the feature information of the grid area where the center point of the target is located.

[0062] The feature maps of the grid area A and each grid area in the group of adjacent grid areas are extracted by a convolutional network, and are compared and selected to obtain the group of adjacent grid areas with the highest degree of association.

[0063] The local attention matching algorithm mainly extracts the feature maps of the grid area A and eight adjacent grid areas by a small convolutional network, and compares and selects the group of adjacent grid areas with a higher degree of association.

[0064] The specific steps of extracting the feature maps are as follows:

[0065] The images of A and eight adjacent grid areas are input respectively, and each grid area feature map is output by 9 3-1 structures, each of which includes a 3x3 convolution and a 1x1 convolution. As shown in Figure 4 .

[0066] Wherein, 3x3 convolution, which extracts the input as the first layer feature map, the main purpose is to extract the key point information in the graph, to provide basic information for the subsequent comparison of different regions; 1 refers to 1x1 convolution, which converts the first layer feature map into the feature map to be output, the main purpose is to integrate and fuse the key point information of different regions, so as to prevent two graphs from being matched to similar regions only because their outlines are similar and other details are different.

[0067] The specific steps of selecting the region group with high correlation degree include:

[0068] The similarity information β(A, B1) of A and B1 is obtained by the following formula:

[0069]

[0070] Wherein A i,j is the pixel value of the i-th row and j-th column of the A square region feature map output, B1 i,j is the pixel value of the i-th row and j-th column of the B1 square region feature map output. A i,j and B1 i,j The value range is [0, 255];

[0071] Similarly, β(A, B2), β(A, B3), and β(A, B4) are obtained.

[0072] Let β(A, B) = [β(A, B1) + β(A, B2) + β(A, B3) + β(A, B4)] / 4, and similarly, β(A, C) and β(A, D) are obtained.

[0073] Compare β(A, B), β(A, C), and β(A, D), and take the square region corresponding to the smallest β value as the adjacent region group with the highest correlation degree.

[0074] Step S33, if the position of the target detected in the next frame does not appear in the A square region but in the adjacent region group with the highest correlation degree, it is considered to be correctly identified and the target is retained; otherwise, the position of the target is corrected. The specific steps of correction include:

[0075] Based on the Euclidean distance between the center point position of the target in the current frame and the next frame multiplied by min(β(A, B), β(A, C), β(A, D)), a correction distance value is obtained, and based on the correction distance value, the center point position of the target in the next frame is still located on the line connecting the center points of the target in the current frame and the next frame; if the center point of the target in the next frame returns to the A region after correction, the target is retained, otherwise, the output confidence level of the target detected in the next frame is combined with the target recognition algorithm to determine whether to retain or discard the target, if the output confidence level is very high, the target is retained, otherwise, it is discarded.

[0076] It should be noted that all the detected targets are subjected to the above judgment, and the target position of the misjudged target is corrected.

[0077] Step S4, based on the front and rear N frame information of the same target output by the target recognition algorithm, the category accuracy of the target is judged by the front and rear frame association algorithm, and the target with inaccurate category judgment is deleted.

[0078] The front and rear N frame information of the same target output by the original algorithm is stored, and the reliability of the target category information output by the original target recognition algorithm is improved by the front and rear frame association algorithm composed of front frame confidence weighted filtering and multi-frame voting. The specific steps include S41-S42.

[0079] Step S41, update the confidence of the current frame according to the following formula:

[0080] P=P i +α i ×P i-1 +α i ×P i-2 +......+α i ×P1

[0081] Where P i is the confidence of the current frame, P i-1 is the confidence of the previous frame, P1 is the first frame confidence, α i is the decay coefficient of the current frame, the range is [0,1], which can be 0.1 as a step through test to get the value that can make the detection result most stable as α i ; if P is greater than the confidence threshold P th , it is considered that the category recognition of the target is accurate, and the output result is output; if P is lower than the confidence threshold P th , the multi-frame voting strategy is executed.

[0082] Step S42, take the detection results of the latest p frames, if q frames of P are greater than the threshold Pth, where q

[0083] Step S5, fuse the corrected target position and the accurate target category with the obtained original video stream picture, and output the recognition result.

[0084] Compared with the prior art, the aviation video stream target recognition algorithm performance improvement method provided by the embodiment solves the problem of poor detection ability of small weak targets and poor robustness in the prior art. The low algorithm power occupation algorithm does not need to increase too much processing time; it is also a plug-and-play algorithm module with strong adaptability. As long as the original video stream image and the position coordinates and category information output by the original recognition algorithm can be obtained, the position accuracy of the original target recognition algorithm can be improved, the false alarm rate can be reduced, the target category recognition accuracy of the original target recognition algorithm can be improved combined with the confidence level, and the target recognition flicker rate of the original target recognition algorithm is reduced.

[0085] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

[0086] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. An aerial video stream target recognition algorithm performance improvement method, characterized in that, The method comprises the following steps: acquiring original video stream images and positions, categories and confidence levels of all targets output by the target recognition algorithm based on the original video stream images; segmenting the original video stream images into multiple squares adaptively based on the positions of all targets and distributing the centers of all targets in different squares as much as possible; the step of segmenting the original video stream images into multiple squares adaptively comprises the following steps: segmenting the original video stream images into MxN squares adaptively, M = [W / w max +k M x δ M ], N = [H / h max +k N x δ N ]; wherein, W, H are the pixel size of the width and height of the output image respectively; w max , h max are the maximum value of the pixel of the width and height of all the targets output by the target recognition algorithm respectively; δ M and δ N are the adaptive coefficients of the width and height respectively; k i is the loop control constant, i takes M or N respectively representing the width and height direction; judging whether the position of the target needs to be corrected based on the correlation degree of the square region where the center point of the target is located and the adjacent square region group, and correcting the position when it is judged that the position needs to be corrected, comprising the following steps: extracting the feature maps of the square region A where the center point of the target is located and each square region in the adjacent square region group through a convolution network, and performing matching and comparison to obtain the adjacent square region group with the highest correlation degree; if the position of the target detected in the next frame does not appear in the A square region but appears in the adjacent square region group with the highest correlation degree, it is considered to be correct recognition, and the target is retained; otherwise, it is judged that the position of the target needs to be corrected; the step of extracting the feature maps of the square region A where the center point of the target is located and each square region in the adjacent square region group through a convolution network comprises the following steps: inputting A and eight adjacent square region images B1, B2, B3, B4, C1, C2, D1 and D2 respectively, and outputting the feature maps of each square region through 9 3-1 structures; each 3-1 structure comprises a 3x3 convolution and a 1x1 convolution; the process of obtaining the adjacent square region group with the highest correlation degree comprises the following steps: the similarity information β(A, B1) of A and B1 square regions is obtained through the following formula: wherein A i，j is the pixel value of the i-th row, j-th column in the A square region feature map, B1 i，j is the pixel value of the i-th row, j-th column in the B1 square region feature map; Similarly, β(A, B2), β(A, B3) and β(A, B4) are obtained. β(A, B) = [β(A, B1) + β(A, B2) + β(A, B3) + β(A, B4)] / 4, and β(A, C) and β(A, D) are obtained in the same way. Compare β(A, B), β(A, C) and β(A, D), and take the square region corresponding to the minimum β value among the three as the adjacent square region group with the highest correlation degree. The adjacent square region group comprises an upper and lower square region group, a left and right square region group and a four-corner square region group adjacent to the square region where the center point of the target is located; there is a partial overlapping region between each two adjacent square regions; the upper and lower square region group comprises two square regions C1 and C2 adjacent to the square region A where the center point of the target is located; the left and right square region group comprises two square regions D1 and D2 adjacent to the square region A; the four-corner square region group comprises four square regions B1, B2, B3 and B4 located at the four corners of the square region A; based on the information of the same target in the previous N frames output by the target recognition algorithm, the category accuracy of the target is judged through a front and back frame correlation algorithm, and the target with inaccurate category judgment is deleted; comprising: updating the confidence level of the current frame according to the following formula: P = P i t + a i x P i-1 + a i x P i-2 +... + a i x P1 wherein P i is the confidence of the current frame, P i-1 is the confidence of the previous frame, P1 is the confidence of the first frame, a i is the attenuation coefficient of the current frame, and the range of a is [0, 1]; if P is greater than a confidence threshold P th , it is considered that the category recognition of the target is accurate. If P is lower than the confidence threshold P th determining whether the class of this target is accurate by performing a multi-frame voting strategy, comprising: Take the detection result of the latest p frames, if the P of q frames is greater than the threshold value P th If q < p, it is determined that the detection result is accurate; if less than q frames meet the requirement, it is determined that the category detection of the target is not accurate, and the target is discarded.

2. The method for improving the performance of the aerial video stream target recognition algorithm according to claim 1, characterized in that, The position correction when judging the need for correction comprises: obtaining a correction distance value based on the Euclidean distance between the center point position of the target in the current frame and the next frame multiplied by the similarity information between the A square region and the square region group where the target is located in the next frame; correcting the center point position of the target in the next frame based on the correction distance value so that the center point position of the target in the next frame is still located on the line connecting the center points of the target in the current frame and the next frame; if the center point of the target in the next frame returns to the A square region after correction, the target is retained, otherwise, the confidence level of the target output by the target recognition algorithm is combined to judge whether the target is retained or discarded.

3. The aerial video stream recognition algorithm performance improvement method of claim 1, wherein, Further comprising: Fusing the corrected target position and the accurate target category with the pictures in the original video stream to output a recognition result.

Images