Object Detection Methods

By preprocessing and sparse sampling of drone aerial videos, generating heat maps and calculating the embed vector distance and offset, the problem of low target detection accuracy in complex drone aerial videos is solved, and the problem of low target detection accuracy in complex environments is realized efficient target recognition.

CN114332734BActive Publication Date: 2024-11-26齐鲁空天信息研究院
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202210009182.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-06
Publication Date
2024-11-26
Estimated Expiration
2042-01-06

AI Technical Summary

Technical Problem

In complex drone aerial photography scenarios, target detection is low due to the uncertainty of the drone's flight altitude and direction, external environment and dynamic background changes, and the recognition accuracy is low, and the long-term target is too small and difficult to detect.

Method used

By preprocessing the target video, dividing it into equal length video units and sparsely sampling, generating the upper left corner, lower right corner and center heat map of the bounding box, calculating the embedding vector distance and offset, filtering the target bounding box, and output retaining bounding box coordinates and categories.

Benefits of technology

It significantly reduces the uncertainty of the target environment, improves the accuracy of target detection in complex scenarios, can identify rotation, scaling, and translation to maintain high accuracy, and detect smaller vision targets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114332734B_ABST
    Figure CN114332734B_ABST
Patent Text Reader

Abstract

The invention discloses a target detection method, comprising: preprocessing a target video, dividing the target video into K video units of equal length; cutting the video units into continuous video frames, and sparsely sampling the video frames to obtain RGB images of the video frames; generating three groups of heat maps according to the continuous RGB images; selecting N upper left corner points, lower right corner points and central key points from the three groups of heat maps respectively according to confidence scores; judging whether the upper left corner point and the lower right corner point belong to the same object by calculating the embedding vector distance between the upper left corner point and the lower right corner point; after detecting that the points belong to the same object, filtering a target bounding box by calculating the offset of the upper left corner point and the lower right corner point; determining whether to retain the target bounding box by detecting the central key point in the target bounding box; and outputting the coordinate information and target category of the retained target bounding box.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer graphics and image processing, and in particular to a target detection method. Background Art

[0002] In recent years, computer graphics and image processing technology has developed rapidly in many fields, and is more evident in public security police work, providing spatiotemporal resource services for major security, traffic management, patrol control and other police operations, and becoming an innovative application of public security police in urban management. With the popularization and widespread application of drone aerial photography technology, massive video data has brought tremendous pressure to manual identification. The use of manual mode to analyze and judge video data, the bottleneck of police personnel's manpower, experience and analytical ability restricts the overall effectiveness of smart police application construction.

[0003] Currently, a common component of object detection is multiple anchor bounding boxes, which are rectangular boxes of various sizes and aspect ratios. They are used as candidate boxes for detection. Predefined anchor bounding boxes are placed on the object to be identified, and the corresponding positions are given by the target values ​​of the ground truth. These methods usually require a large number of anchors to ensure that the predicted bounding boxes have a high intersection-over-union (IoU) value with the ground truth. The size and aspect ratio of the anchors need to be manually designed in advance. In addition, the anchors are often inconsistent with the target values ​​of the ground truth. Only a small part of a large number of anchor bounding boxes can overlap with the ground truth, which creates a huge imbalance between positive and negative samples and slows down the training speed.

[0004] The main technical problems of existing target detection methods include:

[0005] (1) In the case of complex scenes such as drone aerial photography, the existing technology makes the aerial photography target susceptible to changes in the external environment and dynamic background due to the uncertainty of the drone's flight altitude and direction;

[0006] (2) When the direction of the drone changes, the target to be inspected will also undergo image changes such as rotation, scaling and translation, making it difficult to ensure a high recognition accuracy;

[0007] (3) When the altitude of the UAV changes, there may be a situation where the distant target is too small and it is difficult to detect the outline of the target. Summary of the invention

[0008] In order to solve at least one of the above and other technical problems in the prior art, according to an embodiment of one aspect of the present invention, a target detection method is provided, comprising:

[0009] Preprocessing the target video to divide the target video into K video units of equal length, where K is a positive integer greater than 1;

[0010] Dividing the video unit into continuous video frames, and performing sparse sampling on the video frames to obtain RGB images of the video frames;

[0011] Generating three sets of heat maps according to the continuous RGB images, wherein the three sets of heat maps include: a heat map of the upper left corner of the bounding box, a heat map of the lower right corner of the bounding box, and a center heat map;

[0012] Select N upper left corner points, lower right corner points and center key points from the three groups of heat maps respectively according to the confidence scores, where N is a positive integer greater than 1;

[0013] Determining whether the upper left corner point and the lower right corner point belong to the same object by calculating the embedding vector distance between the upper left corner point and the lower right corner point;

[0014] After detecting the same object, filtering the target bounding box by calculating the offset between the upper left corner point and the lower right corner point;

[0015] Determining whether to retain the target bounding box by detecting the central key point in the target bounding box;

[0016] Output the coordinate information of the retained target bounding box and the target category.

[0017] In some embodiments of the present invention, each group of the heat maps has multiple channels, each of the channels is a binary mask for representing the positions of the upper left corner point and the lower right corner point of the channel, the upper left corner point and the lower right corner point having a ground-truth positive position and other positions being negative.

[0018] In some embodiments of the present invention, the intersection over union (IoU) of the bounding box and the ground-truth is used to determine a radius set based on the IoU, and the penalty for negative positions within the radius of the positive position is reduced. The reduction in the penalty for a given radius is determined by an unstandardized 2D Gaussian function, expressed as:

[0019]

[0020] In formula (1), σ is 1 / 3 of the radius, (x, y) is the position information of the target in the RGB image, and the local loss function L is obtained by the unstandardized 2D Gaussian function. det , expressed as:

[0021]

[0022] In formula (2), N is the number of objects in the RGB image, α and β are hyperparameters that control the contribution of each point, and their initial values ​​are set to 2 and 4. is the confidence score of the c-class position (i, j) in the heat map, From formula (1), we can get The Gaussian convex points encoded in Reduce the penalty around the ground-truth.

[0023] In some embodiments of the present invention, when detecting the target, the embedding vector is generated for each detected upper left corner point and the lower right corner point, and the upper left corner point and the lower right corner point are grouped according to the distance of the embedding vector. Specifically, the upper left corner point and the lower right corner point are grouped by using a 1-dimensional embedding upward loss function, and the upper left corner point and the lower right corner point are distinguished by an upward loss function Lp, which is expressed as:

[0024]

[0025] In formula (3), c il is the upper left embedding vector of the target, c jl is the lower right embedding vector of the target, c l c il and c jl The average value of .

[0026] In some embodiments of the present invention, the offsets of the upper left corner point, the lower right corner point and the central key point are calculated by a sampling method. k , expressed as:

[0027]

[0028] In formula (4), x k and k are the x and y coordinates of corner point k, and is the coordinate of the corner point k mapped to the heat map.

[0029] In some embodiments of the present invention, the offset loss function L is calculated offset , expressed as:

[0030]

[0031] In formula (5), L1s is the minimum absolute value deviation, O k is the offset obtained in formula (4), Y is the target value, f(x i ) are estimated values.

[0032] In some embodiments of the present invention, the local loss function L det , the upward loss function Lp and the L1 offset loss function L of the ground-truth offset To calculate the loss function f (loss), expressed as:

[0033]

[0034] In formula (8), is the local loss function of the upper left corner point and the lower right corner point, is the local loss function of the central key point, L p is the upward loss function, is the offset loss function of the upper left corner point and the lower right corner point, is the offset loss function of the central key point, θ and ω are weight factors, and the algorithm is trained by the loss function f(loss).

[0035] In some embodiments of the present invention, the offset of the central key point is used to calculate the embedding vector distance between the upper left corner point and the lower right corner point, and then M candidate boxes are generated, where M is a positive integer greater than 1, and the M candidate boxes are sorted using the offset of the central key point, and the N central key points are selected according to the confidence scores of the M candidate boxes, and the offset of the central key point is used to remap back to the RGB image, and a central area is defined for each candidate box to ensure that the central key point exists in the central area, and at the same time ensure that the category of the central key point is consistent with the category of the candidate box, and the embedding vector distance between the upper left corner point and the lower right corner point is expressed as:

[0036]

[0037] In formula (6), assume that the upper left corner point is (x, y), f tl is the input feature map of the upper left pooling layer, is (i, j)f tl The vector of the position, H is the length of the heat map, from which the embedding vector distance of the lower right corner point is derived as:

[0038]

[0039] In formula (7), f br is the input feature map of the pooling layer in the lower right corner, is the position f at (i, j) br A vector of , W is the width of the heat map;

[0040] The input feature map is pooled by the maximum pooling method of the central pooling layer f tl H j All eigenvectors between (i, j) and (i, H) are called eigenvectors tl ij ;

[0041] The input feature map is pooled by the maximum pooling method of the central pooling layer f br W j All eigenvectors between (i, j) and (W, j) are called eigenvectors br ij ;

[0042] Calculate tl by dynamic programming ij and br ij The embedding vector distance between them is used to determine whether the upper left corner point and the lower right corner point belong to the same object.

[0043] In some embodiments of the present invention, a maximum value on the bounding box is found to determine an internal response value of the bounding box, the information value of the bounding box and the internal visual information value of the target are obtained by adding the maximum value to the internal response value, a central area is set, the central key point in the central area is detected, and the bounding box is retained if the central key point can be detected, and the value of the bounding box is confirmed by the average of the upper left corner point, the lower right corner point and the central key point, and the bounding box is removed if the central key point cannot be detected.

[0044] In some embodiments of the present invention, the bounding box confirms the retention result by detecting the central area, and the formula is:

[0045]

[0046] In formula (9), parameter n is a set odd number, the size of the parameter n determines the size of the central area, and the coordinate position of the upper left corner of the bounding box is set to (tl x , tl y ), the coordinate position of the lower right corner of the bounding box is set to (br x ,br y ), the coordinate position of the upper left corner of the central area (ctl x , ctl y ) and the lower right corner coordinate position (cbr x ,cbr y ), and performing bounding box discrimination by detecting the central key point information in the central area.

[0047] The embodiment of the present invention divides the target video into equal lengths, then performs sparse sampling to output RGB images of a fixed number of frames, and then processes the RGB images. This can solve the problem in the prior art that in complex scenes, the target is easily affected by changes in the external environment and dynamic background due to the uncertainty of the environment, significantly reduces the uncertainty of the target environment, and can quickly and accurately detect the target. By applying the target detection method of the embodiment of the present invention, the heat map of the RGB image is extracted, and then the upper left corner point and the lower right corner point are extracted from the heat map, and then the embedding vectors of the upper left corner point and the lower right corner point can be screened by the target detection method of the embodiment of the present invention. By calculating the embedded vectors, a high recognition accuracy rate can be guaranteed even when the target is rotated, scaled, and translated. By detecting the central key point of the target bounding box, the target bounding box without the central key point is screened, and a smaller target object can be identified. This can effectively solve the problem of the distant target being too small and improve the success rate of detecting the target outline. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The flowchart of the target detection method according to the embodiment of the present invention is schematically shown;

[0049] Figure 2 A schematic diagram of the BaseNet network structure of an embodiment of the present invention is shown;

[0050] Figure 3 The schematic diagram of detecting the central key point of an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0052] But it should be understood that these descriptions are exemplary only, and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, the description of known techniques is omitted to avoid unnecessary confusion of concepts of the present invention.

[0053] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The term "comprising" used herein indicates the existence of features, steps, operations, but does not exclude the existence or addition of one or more other features.

[0054] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0055] Figure 1 The flowchart of the target detection method according to the embodiment of the present invention is schematically shown.

[0056] like Figure 1 As shown, the target detection method includes operations S101 to S108.

[0057] In operation S101 , a target video is preprocessed to divide the target video into K video units of equal length, where K is a positive integer greater than 1.

[0058] According to an embodiment of the present invention, K video units of equal length may be recorded as V={I1, I2, ..., I k The original target video can come from the streaming media server. The monitoring equipment of each drone can remain stationary, with the cruise route as the monitoring scene. The real-time monitoring video is transmitted through the network and stored in the streaming media server. By intercepting and comprehensively processing the target video, the processing efficiency can be effectively improved.

[0059] In operation S102, the video unit is divided into continuous video frames, and the video frames are sparsely sampled to obtain RGB images of the video frames.

[0060] According to an embodiment of the present invention, each uniformly divided video unit can be divided into continuous video frames. The method of sparse sampling of video frames is not limited. For example, sparse sampling can be randomly performed from continuous video frames to obtain an RGB image of a video frame sequence. For a single-scale RGB image test, the resolution of the input original image and the horizontally flipped image is the same as the original resolution; for a multi-scale RGB image test, the resolution of the input original image and the horizontally flipped image can be set to 0.6, 1.2, 1.5, 1.8 and 0.5, 1.0, 1.5, 2.0, respectively. Sparse sampling, also known as compressed sensing, has a wide range of applications in signal processing, information theory, communications and computer vision. The signal is reconstructed by perceiving the continuous signal of the obtained RGB image, arranging the pixels into a matrix, and reconstructing the RGB image. Moreover, the RGB image is not compressed during sparse sampling, thereby ensuring the integrity of the feature set of the image.

[0061] In operation S103, three groups of heat maps are generated according to the continuous RGB images, wherein the three groups of heat maps include: a heat map of the upper left corner of the bounding box, a heat map of the lower right corner of the bounding box, and a center heat map.

[0062] Figure 2 The BaseNet network structure diagram of an embodiment of the present invention is schematically shown.

[0063] According to an embodiment of the present invention, the BaseNet network algorithm can be applied to process the RGB image to generate three sets of heat maps.

[0064] like Figure 2 As shown in the figure, the RGB image is divided into two branches after the BaseNet network algorithm feature extraction. One branch is used to generate the upper left corner heat map and the lower right corner heat map of the bounding box of the target, and the other branch is used to generate the center heat map. The residual module of two 3×3 convolution Conv_BN layers is used to process the upper left corner heat map and the lower right corner heat map of the bounding box, and then the upper left pooling layer and the lower right pooling layer are applied to extract the upper left corner point and the lower right corner point respectively. The 1 3×3 convolution layer is replaced by the center pooling layer to extract the features of the center key point. At this time, the two branches are not connected, and it is impossible to determine whether the upper left corner point, the lower right corner point and the center key point belong to the same target. After adding a 1×1 convolution layer of the center key point, followed by a modified residual block, the key point triple is constructed. The 3×3 convolution Conv_BN_ReLU layer module with 256 channels and 3 Conv_ReLU layer modules are used to predict the heat map, embedding vector distance and offset. The heat map classifies the key points in the upper left corner and the lower right corner, and the embedding vector distance is a set of key points to find a certain target. The same object is determined by judging the similarity of the two sets of embedding vectors. If the distance is less than a certain threshold, it is considered to be the same target. The offset is the bias of the target position calculated based on the coordinates. In addition, the single-stage method removes the extraction process of the region of interest RoI, and directly classifies and regresses on the candidate bounding box. The parameter threshold is gradually increased through a set of detectors to solve the overfitting problem and the image quality mismatch problem in the test. Object detection based on key points avoids the problems caused by multiple regressions of candidate bounding boxes.

[0065] According to an embodiment of the present invention, by using a single-stage keypoint-based detector as a baseline backbone network BaseNet, continuous RGB images are input into the BaseNet network, a RGB image feature set is obtained through the BaseNet network algorithm, and then the feature set is output as a heat map. In addition to the BaseNet network algorithm, it can also be implemented by the VGG16 network algorithm, the VGG19 network algorithm, the residual network algorithm ResNets, the stacked hourglass network algorithm Stacked Hourglass Networks, and the DenseNet network algorithm. The resolution of the RGB image input by the BaseNet network algorithm is 512×512, and the resolution of the three sets of heat map images generated is 128×128.

[0066] In operation S104, N upper left corner points, lower right corner points and center key points are respectively selected from the three groups of heat maps according to the confidence scores, where N is a positive integer greater than 1.

[0067] According to an embodiment of the present invention, the confidence score and the size of N can be determined according to actual needs. For example, the first 60 center key points, the first 60 upper left corner points, and the first 60 lower right corner points can be selected from the heat map to detect the border.

[0068] In operation S105, it is determined whether the upper left corner point and the lower right corner point belong to the same object by calculating the embedding vector distance between the upper left corner point and the lower right corner point.

[0069] According to an embodiment of the present invention, in the process of implementing the present invention, it is found that if the target is judged directly based on the bounding box, it is easy to confuse the bounding boxes of different targets. The embodiment of the present invention first judges the upper left corner point and the lower right corner point to confirm the target object, which can identify the target more accurately.

[0070] In operation S106 , after the detection of the same object, the target bounding box is filtered by calculating the offsets of the upper left corner point and the lower right corner point.

[0071] According to an embodiment of the present invention, by applying the BaseNet network algorithm of an embodiment of the present invention, a heat map of an RGB image is extracted, and then the upper left corner point and the lower right corner point are extracted from the heat map. Then, the offset of the upper left corner point and the lower right corner point can be filtered through the BaseNet network algorithm, and a high recognition accuracy rate can be guaranteed even when the target is rotated, scaled, and translated.

[0072] In operation S107 , it is determined whether to retain the target bounding box by detecting a central key point in the target bounding box.

[0073] According to the embodiments of the present invention, by detecting the key points at the center of the target bounding box, the problem that the distant target is too small can be effectively solved, thereby improving the success rate of detecting the target outline.

[0074] In operation S108 , coordinate information of the retained object bounding box and the object category are output.

[0075] According to the embodiment of the present invention, compared with directly outputting the information of the entire target bounding box in the prior art, the outputted coordinate information of the retained target bounding box and the target category can more accurately identify the target.

[0076] According to an embodiment of the present invention, the target is confirmed by detecting the upper left corner point, lower right corner point and center key point of the target video, rather than by detecting the object bounding box of the target video. This can avoid errors caused by multiple detection regressions in the detection process of the object bounding box, greatly simplify the output of the target bounding box, reduce the extraction of the region of interest ROI, only use the selected upper left corner point, lower right corner point and center key point for prediction, and classify the predicted bounding box.

[0077] According to an embodiment of the present invention, each set of heat maps has multiple channels, each channel is a binary mask for representing the positions of the upper left corner point and the lower right corner point of the channel, the upper left corner point and the lower right corner point have a ground-truth positive position, and the other positions are negative.

[0078] According to an embodiment of the present invention, it is found during the implementation of the present invention that if a pair of negative upper left corner points and lower right corner points are close to their respective ground-truth positions, a bounding box overlapping with the ground-truth is generated.

[0079] According to an embodiment of the present invention, the intersection-over-union (IoU) of the bounding box and the ground-truth is used to determine the radius set according to the IoU, and the penalty for negative positions within the radius of the positive position is reduced. The reduction amount of the penalty for a given radius is determined by a non-standardized 2D Gaussian function, which is expressed as:

[0080]

[0081] In formula (1), σ is 1 / 3 of the radius, (x, y) is the location information of the target in the RGB image, and the local loss function L is obtained by the unstandardized 2D Gaussian function. det , expressed as:

[0082]

[0083] In formula (2), N is the number of objects in the RGB image, d and β are hyperparameters that control the contribution of each point, and their initial values ​​are set to 2 and 4. is the confidence score of the c-class position (i, j) in the heat map, From formula (1), we can get The Gaussian convex points encoded in Reduce the penalty around the ground-truth.

[0084] According to an embodiment of the present invention, IOU≥t, the size of t can be determined according to actual needs. For example, it can be set to 0.75 to determine the size of the target object, thereby determining the radius. The local loss function is one of the important parameters of the BaseNet network algorithm, which can effectively improve the accuracy of the BaseNet network algorithm.

[0085] According to an embodiment of the present invention, when detecting an object, an embedding vector is generated for each detected upper left corner point and lower right corner point, and the upper left corner point and the lower right corner point are grouped according to the distance of the embedding vector. Specifically, the upper left corner point and the lower right corner point are grouped by using a 1-dimensional embedding upward loss function, and the upward loss function L is used. p To distinguish the upper left corner and the lower right corner, it is expressed as:

[0086]

[0087] In formula (3), c il is the upper left embedding vector of the target, c jl is the lower right embedding vector of the target, c l c il and c jl The average value of .

[0088] According to an embodiment of the present invention, an upward loss function L is applied to the upper left corner point and the lower right corner point around the ground-truth. p There may be multiple targets in the RGB image, and multiple upper left corner points and lower right corner points may be detected. The embodiment of the present invention calculates the embedding vector distance by using the embedding vectors of the upper left corner point and the lower right corner point of the BaseNet network algorithm to determine whether the upper left corner point and the lower right corner point are from the same target object.

[0089] According to an embodiment of the present invention, an embedding vector is generated for each detected upper left corner point and lower right corner point when detecting each target, and the upper left corner point and lower right corner point are grouped according to the distance between the embedding vectors. The actual value of the embedding vector is not important. Only the distance between the embeddings is used to group the upper left corner point and lower right corner point. The BaseNet network algorithm predicts the embedding vector of each upper left corner point and lower right corner point. If the upper left corner point and the lower right corner point belong to the same bounding box, the distance between their embedding vectors should be small enough, and then the corner points can be grouped according to the distance between the embedding vectors of the upper left corner point and the lower right corner point.

[0090] According to an embodiment of the present invention, the offsets of the upper left corner point, the lower right corner point and the center key point are calculated by a sampling method. k , expressed as:

[0091]

[0092] In formula (4), x k and k are the x and y coordinates of corner point k, and is the coordinate of the corner point k mapped to the heat map.

[0093] According to an embodiment of the present invention, for the offset O k When full convolution is applied to RGB images, the image size of the output after convolution is usually smaller than the input image. Therefore, the position coordinates (x, y) in the input RGB image are mapped to the positions in the three sets of heat maps as Where n is the downsampling calculation factor. When remapped to the input RGB image, some precision is lost, which affects the intersection-union (IOU) between the predicted bounding box and the ground-truth. To this end, the upper left and lower right corners are slightly adjusted by predicting the position offset, and then remapped to the input RBG image to obtain an offset O. k .

[0094] According to an embodiment of the present invention, the offset loss function L is calculated offset , expressed as:

[0095]

[0096] In formula (5), L1s is the minimum absolute value deviation, Ok is the offset obtained in formula (4), Y is the target value, and f(x i ) are estimated values.

[0097] According to an embodiment of the present invention, the upper left corner points of all categories predicted by the BaseNet network algorithm share a set of offsets, and the lower right corner points of all categories share another set of offsets. The temporally smoothed L1 offset loss function L is used at the upper left corner point and lower right corner point positions predicted by the ground-truth. offset , minimizing the sum of the absolute differences between the target value and the estimated value.

[0098] According to an embodiment of the present invention, the local loss function L det , the upward loss function Lp and the ground-truth L1 offset loss function L offset To calculate the loss function f(loss), expressed as:

[0099]

[0100] In formula (8), is the local loss function of the upper left corner and the lower right corner, is the local loss function of the central key point, Lp is the upward loss function, is the offset loss function of the upper left corner and the lower right corner, is the offset loss function of the center key point, θ and ω are weight factors, and the algorithm is trained through the loss function f(loss).

[0101] According to an embodiment of the present invention, in the process of implementing the present invention, it is found that too high θ and ω will lead to a decrease in the performance of the BaseNet network algorithm. The initial values ​​can be set to 0.1 and 1. Training the BaseNet network algorithm with the loss function f (loss) can make the data model more robust.

[0102] According to an embodiment of the present invention, the offset of the central key point is used to calculate the embedding vector distance between the upper left corner point and the lower right corner point, and then M candidate boxes are generated, where M is a positive integer greater than 1, and the offset of the central key point is used to sort the M candidate boxes, and N central key points are selected according to the confidence scores of the M candidate boxes. The offset of the central key point is used to remap the image back to the RGB image, and a central area is defined for each candidate box to ensure that there is a central key point in the central area, and at the same time ensure that the category of the central key point is consistent with the category of the candidate box. The embedding vector distance between the upper left corner point and the lower right corner point is expressed as:

[0103]

[0104] In formula (6), assume that the upper left corner point is (x, y), f tl is the input feature map of the upper left pooling layer, is (i, j)f tl The vector of the position, H is the length of the heat map, from which the embedding vector distance of the lower right corner point is derived as:

[0105]

[0106] In formula (7), f br is the input feature map of the pooling layer in the lower right corner, is the position f at (i, j) br The input feature map is pooled by the maximum pooling method f through the center pooling layer. tl H j All eigenvectors between (i, j) and (i, H) are called eigenvectors tl ij ; The input feature map passes through the maximum pooling method of the central pooling layer f br W j All eigenvectors between (i, j) and (W, j) are called eigenvectors brij ; Calculate tl through dynamic programming ij and br ij The embedding vector distance between them is used to determine whether the upper left corner point and the lower right corner point belong to the same object.

[0107] According to an embodiment of the present invention, the BaseNet network algorithm downsamples the input features through convolutional layers and maximum pooling layers, and then maps the upsampled features back to the original resolution through upsampling and convolutional layers. Since details are lost in the maximum pooling layer, a skip layer is added to bring the details back to the upsampled features, capturing global and local features in a unified structure.

[0108] According to an embodiment of the present invention, it is found in the process of implementing the present invention that by setting the step size to 2 to reduce the feature resolution, the feature resolution can be reduced by 5 times and the number of feature channels can be increased. When the features are upsampled, two residual modules are applied, followed by a nearest neighbor upsampling. Each jump connection also contains two residual modules, which replace the reverse intermediate prediction by adding intermediate supervision because the latter will damage the performance of the BaseNet network algorithm. To determine whether a pixel is the upper left corner point, it is necessary to calculate the top boundary of the target horizontally to the right and the leftmost boundary of the object vertically to the bottom, and better locate the corner points by encoding explicit prior knowledge.

[0109] According to an embodiment of the present invention, the maximum value on the bounding box is found to determine the internal response value of the bounding box, the information value of the bounding box and the internal visual information value of the target are obtained by adding the maximum value to the internal response value, a central area is set, and the central key point in the central area is detected. If the central key point can be detected, the bounding box is retained, and the value of the bounding box is confirmed by the average of the upper left corner point, the lower right corner point and the central key point. If the central key point cannot be detected, the bounding box is removed.

[0110] According to an embodiment of the present invention, it is found in the process of implementing the present invention that the upper left corner point and the lower right corner point are usually located on the bounding box of the target. Due to the lack of local appearance features, the information of the geometric center of the object does not necessarily have a high degree of recognition. For example, the human head has a strong visual pattern, but the geometric center is usually in the middle of the human body. The embodiment of the present invention eliminates the wrong bounding box by using the central key point position detected by the BaseNet network algorithm, determines the upper left corner point and the lower right corner point by finding the maximum value of the bounding box, determines the visual pattern inside the target by the upper left corner point and the lower right corner point, and then finds the maximum value of the target in the horizontal and vertical directions. For example, the maximum value can be found along the bounding box, and the internal maximum response value can be found from the position of the maximum value to the inside of the bounding box, and the two maximum values ​​are added together, so that the central key point can obtain both the information of the bounding box and the visual information inside the object.

[0111] According to an embodiment of the present invention, the bounding box confirms the retention result by detecting the central area, and the formula is:

[0112]

[0113] In formula (9), parameter n is a set odd number. The size of parameter n determines the size of the central area. The coordinate position of the upper left corner of the bounding box is set to (tl x , tl y ), the coordinate position of the lower right corner of the bounding box is set to (br x ,br y ), the coordinate position of the upper left corner of the center area (ctl x , ctl y ) and the lower right corner coordinate position (cbr x ,cbr y ), and bounding box discrimination is performed by detecting the central key point information in the central area.

[0114] Figure 3 The schematic diagram of detecting the central key point of an embodiment of the present invention is schematically shown.

[0115] like Figure 3 As shown, (tl x , tl y ) is the coordinate position of the upper left corner of the bounding box, (br x ,br y ) is the coordinate position of the lower right corner of the bounding box, (ctl x , ctl y ) is the coordinate position of the upper left corner of the center area, (cbr x ,cbr y) is the coordinate position of the lower right corner of the center area. As the parameter n increases, the coordinate point (tl x , tl y ) and (br x ,br v ) remains unchanged, through the coordinate point (ctl x , ctl y ) and (cbr x ,cbr y ) is reduced. For example, when n=3, the target's central key point is detected as a 3×3 table, and the number of candidate bounding boxes is less than or equal to 24. When n=5, the target's central key point is detected as a 5×5 table, and the number of candidate bounding boxes is less than or equal to 60. According to formula (9), an adaptive central area can be obtained, and then the central area is detected to see whether it contains the central key point to perform bounding box discrimination. The size of the central area in the bounding box will affect the detection result. For example, a smaller central area will result in a low recall rate of a small border, and a larger central area will result in a low precision of a large border. The adaptive central area generates a larger central area for a small border and a smaller central area for a large border.

[0116] So far, the embodiments of the present invention have been described in detail in conjunction with the accompanying drawings. It should be noted that the implementation methods not shown or described in the drawings or the body of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each component are not limited to the various specific structures, shapes or methods mentioned in the embodiments, and ordinary technicians in the field can simply change or replace them.

[0117] It should also be noted that, in the specific embodiments of the present invention, unless otherwise indicated, the numerical parameters in this specification and the attached claims are approximate values ​​and can be changed according to the desired characteristics obtained through the content of the present invention. Specifically, all numbers used in the specification and claims to express the size, range conditions, etc. of the composition should be understood to be modified by the term "about" in all cases. In general, the meaning of the expression is to include a change of ±10% in some embodiments, a change of ±5% in some embodiments, a change of ±1% in some embodiments, and a change of ±0.5% in some embodiments by a specific number.

[0118] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0119] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target detection method, include: Preprocessing the target video, dividing the target video into K video units of equal length, where K is a positive integer greater than 1; Dividing the video unit into continuous video frames, and performing sparse sampling on the video frames to obtain RGB images of the video frames; Generating three sets of heat maps according to the continuous RGB images, wherein the three sets of heat maps include: a heat map of the upper left corner of the bounding box, a heat map of the lower right corner of the bounding box, and a center heat map; Select N upper left corner points, lower right corner points and center key points from the three groups of heat maps respectively according to the confidence scores, where N is a positive integer greater than 1; Determining whether the upper left corner point and the lower right corner point belong to the same object by calculating the embedding vector distance between the upper left corner point and the lower right corner point; After detecting the same object, filtering the target bounding box by calculating the offset between the upper left corner point and the lower right corner point; Determining whether to retain the target bounding box by detecting the central key point in the target bounding box; Output the coordinate information of the retained target bounding box and the target category.

2. According to the method of claim 1, each group of the heat maps has multiple channels, each of the channels is a binary mask for representing the positions of the upper left corner point and the lower right corner point of the channel, the upper left corner point and the lower right corner point have a ground-truth positive position, and the other positions are negative.

3. According to the method of claim 2, the intersection over union (IoU) of the bounding box and the ground-truth is used to determine a radius set according to the IoU, and the penalty for negative positions within the radius of the positive position is reduced. The reduction amount of the penalty for a given radius is determined by an unstandardized 2D Gaussian function, expressed as: In formula (1), σ is 1 / 3 of the radius, (x, y) is the position information of the target in the RGB image, and the local loss function L is obtained by the unstandardized 2D Gaussian function. det , expressed as: In formula (2), N is the number of objects in the RGB image, α and β are hyperparameters that control the contribution of each point, and their initial values ​​are set to 2 and 4. is the confidence score of the c-class position (i, j) in the heat map, From formula (1), we can get The Gaussian convex points encoded in Reduce the penalty around the ground-truth.

4. The method according to claim 3, generating the embedding vector for each detected upper left corner point and the lower right corner point when detecting the target, and grouping the upper left corner point and the lower right corner point according to the distance of the embedding vector, specifically by using a 1-dimensional embedding upward loss function to group the upper left corner point and the lower right corner point, and using an upward loss function L p To distinguish the upper left corner point and the lower right corner point, it is expressed as: In formula (3), c il is the upper left embedding vector of the target, c jl is the lower right embedding vector of the target, c l c il and c jl The average value of .

5. The method according to claim 4, wherein the offsets of the upper left corner point, the lower right corner point and the central key point are calculated by a sampling method. k , expressed as: In formula (4), x k and k are the x and y coordinates of corner point k, and is the coordinate of the corner point k mapped to the heat map.

6. According to the method of claim 5, calculating the offset loss function L offset , expressed as: In formula (5), L 1 s is the minimum absolute deviation, O k is the offset obtained in formula (4), Y is the target value, f(x i ) are estimated values.

7. The method according to claim 6, wherein the local loss function L det , the upward loss function Lp and the L1 offset loss function L of the ground-truth offset To calculate the loss function f(loss), expressed as: In formula (8), is the local loss function of the upper left corner point and the lower right corner point, is the local loss function of the central key point, L p is the upward loss function, is the offset loss function of the upper left corner point and the lower right corner point, is the offset loss function of the central key point, θ and ω are weight factors, and the algorithm is trained by the loss function f(loss).

8. According to the method of claim 7, the offset of the central key point is used to calculate the embedding vector distance between the upper left corner point and the lower right corner point, and then M candidate boxes are generated, where M is a positive integer greater than 1, the offset of the central key point is used to sort the M candidate boxes, the N central key points are selected according to the confidence scores of the M candidate boxes, and the offset of the central key point is used to remap back to the RGB image, a central area is defined for each candidate box, and the central key point is ensured to exist in the central area, and the category of the central key point is ensured to be consistent with the category of the candidate box, and the embedding vector distance between the upper left corner point and the lower right corner point is expressed as: In formula (6), assume that the upper left corner point is (x, y), f tl is the input feature map of the upper left pooling layer, is (i, j)f tl The vector of the position, H is the length of the heat map, from which the embedding vector distance of the lower right corner point is derived as: In formula (7), f br is the input feature map of the pooling layer in the lower right corner, is the position f at (i, j) br A vector of , W is the width of the heat map; The input feature map is pooled by the maximum pooling method of the central pooling layer f tl H j All eigenvectors between (i, j) and (i, H) are called eigenvectors tl ij ; The input feature map is pooled by the maximum pooling method of the central pooling layer f br W j All eigenvectors between (i, j) and (Wj) are called eigenvectors br ij ; Calculate tl by dynamic programming ij and br ij The embedding vector distance between them is used to determine whether the upper left corner point and the lower right corner point belong to the same object.

9. The method according to claim 1, wherein the maximum value on the bounding box is found to determine the internal response value of the bounding box, the information value of the bounding box and the internal visual information value of the target are obtained by adding the maximum value to the internal response value, a central area is set, the central key point in the central area is detected, the bounding box is retained if the central key point can be detected, the value of the bounding box is confirmed by the average of the upper left corner point, the lower right corner point and the central key point, and the bounding box is removed if the central key point cannot be detected.

10. The method according to claim 9, wherein the bounding box confirms the retention result by detecting the central area, and the formula is: In formula (9), parameter n is a set odd number, the size of the parameter n determines the size of the central area, and the coordinate position of the upper left corner of the bounding box is set to (tl x , tl y ), the coordinate position of the lower right corner of the bounding box is set to (br x ,br y ), the coordinate position of the upper left corner of the central area (ctl x , ctl y ) and the lower right corner coordinate position (cbr x ,cbr y ), and performing bounding box discrimination by detecting the central key point information in the central area.

Citation Information

Cited By

  • Video target batch asynchronous detection method and system based on historical video recording

    CN123020771A