Pedestrian detection system for dense scene and construction method thereof
By introducing backbone network, decoupling prediction module and suppression detection recall module into the pedestrian detection system, the missed detection and misdetection problems caused by occlusion in dense scenes are solved, the accuracy and performance of pedestrian detection are improved, and the occlusion of pedestrians is enhanced Perception of the part.
Patent Information
- Application Number
- CN202410848981.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-12-27
AI Technical Summary
The existing pedestrian detection technology has serious problems of missed and missed detection due to occlusion in dense scenes, making it difficult to effectively distinguish the complex relationship between pedestrians and backgrounds, especially in dense scenes, where the high overlap and occlusion between pedestrians cause detection difficulties.
A pedestrian detection system for dense scenes is designed, including a backbone network, a decoupled prediction module, a full-body frame visibility prediction module and a suppression detection recall module. By extracting deep semantic features, decomposing pedestrian visual area and whole-body area features, Enhance the perception of occluded pedestrians, and use similarity to determine the suppressed detection box to improve detection performance.
It effectively improves the accuracy and performance of pedestrian detection in dense scenarios, reduces missed and missed detection, enhances the ability to distinguish pedestrian parts, and can be combined with most target detection frameworks to stably improve the detection performance of the detector.
Smart Images
Figure CN119206598A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a pedestrian detection system for dense scenes and a construction method thereof. Background Art
[0002] In recent years, with the rapid development of computer hardware and related theories, artificial intelligence technology has been widely used. As an important part of artificial intelligence technology, computer vision technology has attracted a large number of researchers to devote themselves to the field of computer vision. Computer vision is an interdisciplinary field that studies how to make computer systems understand and interpret the content of digital images or videos. It involves extracting information from images or videos, effectively processing this information and serving related tasks.
[0003] Pedestrian detection is an important research topic in the field of computer vision. Its goal is to accurately locate the position information of pedestrians in images or videos. The basic model of pedestrian detection is derived from the existing target detection model. It is specially designed in the field of target detection to adapt to the complex background, posture and other factors of pedestrians. At present, pedestrian detection technology is widely used in scenarios such as intelligent monitoring and autonomous driving. At the same time, as a basic task of computer vision, pedestrian detection technology can serve key areas such as pedestrian target tracking and pedestrian re-identification.
[0004] However, pedestrian detection is generally applied in densely populated scenes such as subway stations, pedestrian streets, and intersections. These scenes have a high pedestrian density and a complex background, which often leads to intra-class and inter-class occlusion problems between pedestrians and the background. On the one hand, occlusion will cause the loss of pedestrian appearance features, making it difficult for the detector to detect pedestrians. On the other hand, occlusion causes a high degree of overlap of pedestrians in space, making it difficult for existing detectors to distinguish between redundant detections and highly overlapping real detections. Combining these two difficulties, occlusion will cause relatively serious missed detections and false detections. Therefore, the problem of pedestrian detection is a key difficulty in the field of computer vision at this stage. Summary of the invention
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] The present invention provides a pedestrian detection system for dense scene, the system comprising:
[0007] Backbone network, used to extract deep semantic features from images;
[0008] The decoupled prediction module includes a pedestrian visible region extraction network and a full body region extraction network, which are used to obtain the pedestrian's visible region candidate frame and full body region candidate frame through the pedestrian's visible region extraction network and the full body region extraction network, respectively, and perform classification and regression calculations on them to obtain the full body detection frame, the full body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence;
[0009] The full-body frame visibility prediction module divides the pedestrian's full-body frame into five parts according to the body parts, and enhances the network's perception of occluded pedestrians by predicting the visibility of the five parts;
[0010] The suppressed detection recall module includes a similarity discrimination network and a suppressed detection recall strategy. The similarity discrimination network consists of three fully connected layers. The suppressed detection recall strategy is used to recall the full body frames that should have been retained but were suppressed due to the high overlap between pedestrians.
[0011] The detection method is:
[0012] (1) Input the image to be detected;
[0013] (2) Obtain the deep semantic features of the image to be detected through the backbone network;
[0014] (3) The deep semantic features are used to extract the visible region candidate boxes and the whole body region candidate boxes through the visible region extraction network and the whole body region extraction network;
[0015] (4) The visible region candidate frame and the whole body region candidate frame are subjected to classification and regression calculation to obtain the whole body detection frame, the whole body detection frame classification confidence, the visible detection frame, and the visible detection frame classification confidence;
[0016] (5) The full body detection frame, the full body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence suppress detection recall to obtain the image detection result.
[0017] Furthermore, the specific process of step (5) is as follows:
[0018] (51) After processing multiple images, a full-body detection frame set B, a full-body detection frame classification confidence set S, a visible detection frame set H, and a visible detection frame classification confidence set J are obtained, and the full-body detection frame set of the pedestrian to be recalled is set to R, where R is an empty set;
[0019] (52) performing non-maximum suppression on the whole body detection frame set B to obtain a retained whole body detection frame set B1 and a suppressed whole body detection frame set B2, and arranging B1 and B2 in descending order according to classification confidence;
[0020] (53) Perform non-maximum suppression on the visible detection box set H to obtain the retained visible detection box set H1, and arrange H1 in descending order according to classification confidence;
[0021] (54) Take a visible box h from H1 in descending order according to the classification confidence j , calculate all full body boxes and h in B1 j The area intersection ratio IOF is calculated, and the whole body frame whose area intersection ratio IOF is greater than the threshold value 0.6 is taken out;
[0022] (55) Extract the features of the corresponding area in the full-body frame whose area intersection ratio IOF is greater than the threshold 0.6 in step (54), and calculate the full-body frame and visible frame h obtained by similarity discrimination network j If the similarity of the corresponding regional features exceeds a certain threshold beta = 0.2, h is deleted from H1 j ;
[0023] (56) Take a visible box h from H1 in descending order according to the classification confidence j , repeat steps (54) and (55) until all visible frames in H1 have executed steps (54) and (55);
[0024] (57) Take a visible box h from H1 after elimination in descending order according to the classification confidence j , calculate the visible frame h j The similarity between the features of the region corresponding to the full-body frame in R, if the similarity exceeds the threshold beta=0.2, then repeat step (57); otherwise, proceed to step (58);
[0025] (58) Take out the whole body detection box b in B2 in descending order of classification confidence i , calculate b i and h j IOF, if the foreground intersection value is less than the threshold 0.6, and b is calculated according to the similarity discrimination network i Corresponding regional features and h j If the similarity of the corresponding regional features is less than the threshold γ = 0.7, then b i Put it into R and return to step (57);
[0026] (59) The final set of full-body detection frames R of the pedestrians to be recalled and the set of full-body detection frames B1 retained after non-maximum suppression are merged to obtain the final image detection result.
[0027] Furthermore, all the full-body frames and visible frames h in the retained full-body detection frame set B1 in step (54) j The area intersection ratio IOF refers to the area of the whole body frame and the visible frame h in B1. j The area of the intersection divided by the visible box hj The specific formula is:
[0028]
[0029] The present invention provides a method for constructing a pedestrian detection system for dense scenes, the method for constructing is used for the above detection system, and the method for constructing comprises:
[0030] Step 1, obtaining a training sample set, the training sample set including a plurality of training samples, each of the training samples including an image containing a pedestrian and a corresponding visible frame annotation and a full body frame annotation;
[0031] Step 2: extracting deep semantic features of training samples through the backbone network, and training the visible area extraction network and the whole body area extraction network through the deep semantic features so that they can extract visible area candidate frames and whole body area candidate frames;
[0032] Then, classification and regression training is performed on the visible region candidate frame and the whole body region candidate frame, and the visible detection frame and the whole body detection frame are obtained through classification and regression, and the regression loss function of the visible region candidate frame and the whole body region candidate frame obtained through classification and regression is calculated;
[0033] Step 3, training the visibility network of the five parts of the human body according to the visible detection frame and the whole body detection frame obtained by the training of the training samples and the deep semantic features extracted by the backbone network, so as to enhance the visibility of the five parts of the human body, enhance the network's perception of occluded pedestrians, and calculate the enhanced loss function of the enhanced visibility of the five parts of the human body;
[0034] Step 4: By optimizing the regression loss function and the enhancement loss function, their hyperparameters are adjusted until the total loss is minimized to obtain the detection model;
[0035] Step 5: Freeze the detection model. Based on the visual detection frame and full-body detection frame obtained through training of the training samples, sample the corresponding features of the overlapping visual detection frame and full-body detection frame as training samples, train the similarity discrimination network and the recall strategy of suppression detection, so that it outputs the similarity of the features in the full-body frame and the visual frame, and recalls the full-body detection frame that should have been retained but was suppressed due to the high overlap between pedestrians.
[0036] Furthermore, the process of training the visible area extraction network and the whole body area extraction network in step 2 is as follows:
[0037] Considering the anchor frames existing in the whole body region extraction network, calculating the intersection-and-union ratio of the current anchor frame and the whole body annotation frame in each training sample, and taking the whole body annotation frame with the largest intersection-and-union ratio with the current anchor frame as the assigned label of the current whole body region extraction network;
[0038] Considering the visible annotation box corresponding to the whole body annotation box, determine whether the ratio of the area of the visible annotation box in the current anchor box to the entire visible annotation box reaches a threshold, and use whether the threshold is reached as the allocation standard of the current anchor box with respect to the visible annotation box corresponding to the whole body annotation box;
[0039] If the current anchor frame has an assigned full-body annotation frame, and the current anchor frame meets the assignment criteria of the full-body annotation frame corresponding to the visible annotation frame, then the current anchor frame is assigned to the full-body annotation frame;
[0040] Considering the anchor boxes existing in the visible area extraction network, the intersection-and-union ratio of the current anchor box and each visible annotation box is calculated, and the visible annotation box with the largest intersection-and-union ratio with the current anchor box is used as the intersection-and-union ratio label of the current anchor box;
[0041] Calculate the intersection-and-union ratio of the current whole-body region candidate frame and each whole-body annotation frame, and use the whole-body annotation frame with the largest intersection-and-union ratio with the current whole-body region candidate frame as the intersection-and-union ratio assignment label of the current whole-body region candidate frame; at the same time, calculate the intersection-and-union ratio of the current visible region candidate frame and each visible annotation frame, and use the visible annotation frame with the largest intersection-and-union ratio with the current visible region candidate frame as the intersection-and-union ratio assignment label of the current visible region candidate frame;
[0042] Iteratively train the visible region extraction network and the whole body region extraction network. After each training, calculate the regression loss of the whole body region candidate box and the regression loss of the visible region candidate box.
[0043] Based on the regression loss function of the classification and regression of the visible area candidate boxes and the whole body area candidate boxes, the parameters of the backbone network, the visible area extraction network and the whole body area extraction network are adjusted.
[0044] Furthermore, the anchor frame assignment label in the whole body region extraction network satisfies the following formula:
[0045]
[0046] in, Extract the anchor box in the network for the i-th whole body region, Mark the whole body with a box. is the intersection over union (IoU) function, N is the number of full body annotation boxes, and IoU() is the intersection over union (IoU) function;
[0047] The anchor frame in the whole body region extraction network must also satisfy the following formula:
[0048]
[0049]
[0050] in, is the visible annotation box corresponding to the full-body annotation box with the largest IOU, θ is the given judgment threshold, which can be 0.7, For calculation exist The area in Function of area ratio;
[0051] The anchor box assignment label in the visual area extraction network satisfies the following formula:
[0052]
[0053] in, Extract the anchor box of the network for the i-th visible area, is a visible annotation box, is the intersection over union (IoU) function, N is the number of visible annotation boxes, and IoU() is the intersection over union (IoU) function;
[0054] The whole body region candidate box assignment label satisfies the following formula:
[0055]
[0056] in, is the i-th whole body region candidate box;
[0057] The candidate box assignment labels in the visible area satisfy the following formula:
[0058]
[0059] in, is the candidate box of the i-th visible area.
[0060] Furthermore, the calculation formula of the regression function of the anchor box in the whole body region extraction network is:
[0061]
[0062] The calculation formula of the regression loss function of the whole body detection frame obtained by regressing the whole body region candidate frame is:
[0063]
[0064] in, is the i-th whole body region candidate box assigned as a positive sample, Assign labels to the whole body annotation box, K1 is the number of positive samples in the whole body region extraction network stage, and M1 is the number of positive samples in the whole body region candidate box regression stage;
[0065] The calculation formula of the regression function of the anchor box in the visual area extraction network is:
[0066]
[0067] The regression loss function of the visible area candidate box after regression to obtain the visible detection box is calculated as:
[0068]
[0069] in, is the i-th visible area candidate box assigned as a positive sample, Assign labels to the visible annotation boxes, K2 is the number of positive samples in the visible area extraction network stage, and M2 is the number of positive samples in the visible area candidate box regression stage.
[0070] Furthermore, the visibility network includes five independent sub-networks, each sub-network includes two fully connected layers connected in series, and the visibility network training process is:
[0071] The pedestrian's whole body is divided into five parts: head, left and right upper body, left and right lower body. The true visibility of each part (head, left and right upper body, left and right lower body) is calculated based on the whole body region candidate frame and visible frame annotation, specifically:
[0072]
[0073] Use a hard-coded form to convert the actual visibility to a visibility label:
[0074]
[0075] Where x represents the calculated true visibility result, and y represents the hard-coded converted visibility label;
[0076] Train a subnetwork for each body part and calculate the average loss for the five parts:
[0077]
[0078] Among them, y j It represents the visibility label of the jth part, p j Represents the visibility prediction result of the jth part. During the reasoning process, the visible whole body region extraction module branch is discarded to avoid increasing the reasoning cost.
[0079] Furthermore, the similarity discrimination network includes two fully connected layers connected in series, the number of intermediate channels of the two fully connected layers is 1024, and the similarity result is output through the Softmax function;
[0080] The training process of the similarity discrimination network and the recall strategy of suppression detection is as follows:
[0081] First, the detector is frozen, and the detector parameters are not updated during the training of the similarity discrimination network. Then, the visible region candidate box and the whole body region candidate box obtained by the visible region extraction network and the whole body region extraction network are output, and non-maximum suppression is performed on the visible region candidate box and the whole body region candidate box. The area intersection ratio (IOF) of the visible region candidate box and all the whole body region candidate boxes after non-maximum suppression is calculated. If the area intersection ratio (IOF) is greater than the threshold (0.6), it is considered to be a pair of samples.
[0082] Assign a positive label to each pair of examples:
[0083]
[0084] Among them, B v is the visual detection box result output by the detector, B f It is the full body box result output by the detector.
[0085] Except for the positive samples, all other samples are negative samples;
[0086] Based on the definition of positive and negative samples, the training samples are collected, and two memories are designed to store the features of the regions of interest corresponding to the positive and negative samples respectively, and the memories are continuously updated according to the first-in-first-out principle;
[0087] Based on the stored features, a fixed ratio of positive and negative samples are sampled each time for training. The label of the positive sample is 1 and the label of the negative sample is 0.
[0088] Furthermore, the loss function of the similarity discrimination network is:
[0089] L sim =L BCE (l pred ,l sim );
[0090] Among them, l pred is the similarity result output by the similarity discrimination network, l sim is the similarity label of positive and negative samples, L BCE Refers to the binary cross entropy loss function.
[0091] The present invention has the following beneficial effects:
[0092] (1) Based on the existing object detection framework, the present invention combines multiple modules designed specifically for pedestrian detection to form a pedestrian detection solution, which effectively improves the ability of pedestrian detection in dense scenes;
[0093] (2) The present invention utilizes the pedestrian full body frame visibility prediction module during the training process to implicitly align the full body region features with the visible region features, thereby improving the model's ability to distinguish the occluded parts of pedestrians;
[0094] (3) The present invention constructs a suppressed detection recall module, takes the visible detection frame and the full body detection frame of the decoupled prediction module as input, uses the pedestrian visible area features as a query, and determines and recalls the suppressed full body frame detection results by calculating the similarity between the query and the full body area features;
[0095] (4) The present invention can be combined with most proposal-based object detection frameworks to stably improve the detection performance of the detector. It can also be embedded in most proposal-based and various non-maximum suppression-based detectors. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] Figure 1 It is a schematic diagram of the system module of the present invention.
[0097] Figure 2 It is a flow chart of the detection method of the present invention.
[0098] Figure 3 It is a flow chart of obtaining image detection results by suppressing detection recall based on body detection frame, whole body detection frame classification confidence, visible detection frame and visible detection frame classification confidence in the present invention. DETAILED DESCRIPTION
[0099] The specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be pointed out that the embodiments are only specific explanations of the invention and should not be regarded as limitations of the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solutions of the present invention. The protection scope of the present invention shall still be based on the scope defined by the claims.
[0100] Example 1
[0101] like Figure 1 As shown, this embodiment provides a pedestrian detection system for dense scenes, the system comprising:
[0102] Backbone network, used to extract deep semantic features from images;
[0103] The decoupled prediction module includes a pedestrian visible region extraction network and a full body region extraction network, which are used to obtain the pedestrian's visible region candidate frame and full body region candidate frame through the pedestrian's visible region extraction network and the full body region extraction network, respectively, and perform classification and regression calculations on them to obtain the full body detection frame, the full body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence;
[0104] The full-body frame visibility prediction module divides the pedestrian's full-body frame into five parts according to the body parts, and enhances the network's perception of occluded pedestrians by predicting the visibility of the five parts; the full-body frame visibility prediction module only assists in optimizing the network during training;
[0105] The suppressed detection recall module includes a similarity discrimination network and a suppressed detection recall strategy, wherein the similarity discrimination network is composed of three fully connected layers, and the suppressed detection recall strategy is used to recall the full body frames that should have been retained but were suppressed due to the high overlap between pedestrians.
[0106] like Figure 2 As shown, the detection method is:
[0107] S1, input the image to be detected;
[0108] S2, obtains the deep semantic features of the image to be detected through the backbone network;
[0109] S3, deep semantic features are used to extract visible area candidate boxes and full body area candidate boxes through the visible area extraction network and the full body area extraction network;
[0110] S4, the visible area candidate frame and the whole body area candidate frame are classified and regressed to obtain the whole body detection frame, the whole body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence;
[0111] S5, the whole body detection frame, the whole body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence suppress detection recall to obtain the image detection result;
[0112] like Figure 3 As shown, the specific process is:
[0113] S51, after processing multiple images, a full body detection frame set B, a full body detection frame classification confidence set S, a visible detection frame set H and a visible detection frame classification confidence set J of the image to be detected are obtained, and the full body detection frame set of the pedestrian to be recalled is set to R, where R is an empty set;
[0114] S52, performing non-maximum suppression on the whole body detection frame set B to obtain a retained whole body detection frame set B1 and a suppressed whole body detection frame set B2, and arranging B1 and B2 in descending order according to classification confidence;
[0115] S53, performing non-maximum suppression on the visible detection box set H to obtain a retained visible detection box set H1, and arranging H1 in descending order according to classification confidence;
[0116] S54, extract a visible box h from H1 in descending order according to the classification confidence j , calculate all full body boxes and h in B1 jThe area intersection ratio IOF is calculated, and the whole body frame whose area intersection ratio IOF is greater than the threshold value 0.6 is taken out;
[0117] All full-body frames and visible frames h in the retained full-body detection frame set B1 j The area intersection ratio IOF refers to the area of the whole body frame and the visible frame h in B1. j The area of the intersection divided by the visible box h j The specific formula is:
[0118]
[0119] S55, extract the features of the corresponding area in the full-body frame whose area intersection ratio IOF is greater than the threshold value 0.6 in step S54, and calculate the full-body frame and the visual frame h obtained by similarity discrimination network. j If the similarity of the corresponding regional features exceeds a certain threshold beta = 0.2, h is deleted from H1 j ;
[0120] S56, extract a visible box h from H1 in descending order according to the classification confidence j , repeat steps S54 and S55 until all visible frames in H1 have executed steps S54 and S55;
[0121] S57, extract a visible frame h from H1 after elimination according to the classification confidence in descending order j , calculate the visible frame h j The similarity between the feature of the region corresponding to the whole body frame in R, if the similarity exceeds the threshold beta=0.2, then step S57 is repeated; otherwise, step S58 is entered;
[0122] S58, extract the whole body detection box b in B2 in descending order of classification confidence i , calculate b i and h j IOF, if the foreground intersection value is less than the threshold 0.6, and b is calculated according to the similarity discrimination network i Corresponding regional features and h j If the similarity of the corresponding regional features is less than the threshold γ = 0.7, then b i Put it into R, and return to step S57;
[0123] S59, merging the final pedestrian full body detection frame set R to be recalled and the full body detection frame set B1 retained after non-maximum suppression to obtain the final image detection result.
[0124] Example 2
[0125] This embodiment provides a method for constructing a pedestrian detection system for dense scenes, the method comprising:
[0126] Step 1, obtain a training sample set, the training sample set includes multiple training samples, each of the training samples includes an image containing a pedestrian and a corresponding visible box annotation and a full body box annotation; for the training process of the entire detection model, first, obtain a training sample set, wherein the training sample set includes multiple training samples, each of the training samples includes an image containing a pedestrian, and a visible annotation box and a full body annotation box for the image.
[0127] Step 2: extracting deep semantic features of training samples through the backbone network, and training the visible area extraction network and the whole body area extraction network through the deep semantic features so that they can extract visible area candidate frames and whole body area candidate frames;
[0128] Then, classification and regression training is performed on the visible region candidate frame and the whole body region candidate frame, and the visible detection frame and the whole body detection frame are obtained through classification and regression, and the regression loss function of the visible region candidate frame and the whole body region candidate frame obtained through classification and regression is calculated;
[0129] During the training process, for each training sample, first, the deep semantic features in the image are extracted based on the backbone network, and then the deep semantic features are input into the visible area extraction network and the whole body area extraction network, and multiple corresponding candidate boxes are output. Then, the candidate boxes are regressed again based on the high-level features to obtain the visible detection box and the whole body detection box.
[0130] Wherein, considering the anchor frames existing in the whole body region extraction network, calculating the intersection-and-union ratio of the current anchor frame and the whole body annotation frame in each training sample, and taking the whole body annotation frame with the largest intersection-and-union ratio with the current anchor frame as the assigned label of the current whole body region extraction network;
[0131] Considering the visible annotation box corresponding to the whole body annotation box, determine whether the ratio of the area of the visible annotation box in the current anchor box to the entire visible annotation box reaches a threshold, and use whether the threshold is reached as the allocation standard of the current anchor box with respect to the visible annotation box corresponding to the whole body annotation box;
[0132] If the current anchor frame has an assigned full-body annotation frame, and the current anchor frame meets the assignment criteria of the full-body annotation frame corresponding to the visible annotation frame, then the current anchor frame is assigned to the full-body annotation frame;
[0133] Considering the anchor boxes existing in the visible area extraction network, the intersection-and-union ratio of the current anchor box and each visible annotation box is calculated, and the visible annotation box with the largest intersection-and-union ratio with the current anchor box is used as the intersection-and-union ratio label of the current anchor box;
[0134] Calculate the intersection-and-union ratio of the current whole-body region candidate frame and each whole-body annotation frame, and use the whole-body annotation frame with the largest intersection-and-union ratio with the current whole-body region candidate frame as the intersection-and-union ratio assignment label of the current whole-body region candidate frame; at the same time, calculate the intersection-and-union ratio of the current visible region candidate frame and each visible annotation frame, and use the visible annotation frame with the largest intersection-and-union ratio with the current visible region candidate frame as the intersection-and-union ratio assignment label of the current visible region candidate frame;
[0135] Iteratively train the visible region extraction network and the whole body region extraction network. After each training, calculate the regression loss of the whole body region candidate box and the regression loss of the visible region candidate box.
[0136] Based on the regression loss function of the classification and regression of the visible area candidate boxes and the whole body area candidate boxes, the parameters of the backbone network, the visible area extraction network and the whole body area extraction network are adjusted.
[0137] The anchor box assignment label in the whole body region extraction network satisfies the following formula:
[0138]
[0139] in, Extract the anchor box in the network for the i-th whole body region, Mark the whole body with a box. is the intersection over union (IoU) function, N is the number of full body annotation boxes, and IoU() is the intersection over union (IoU) function;
[0140] The anchor frame in the whole body region extraction network must also satisfy the following formula:
[0141]
[0142]
[0143] in, is the visible annotation box corresponding to the full-body annotation box with the largest IOU, θ is the given judgment threshold, which can be 0.7, For calculation exist The area in Function of area ratio;
[0144] The anchor box assignment label in the visual area extraction network satisfies the following formula:
[0145]
[0146] in, Extract the anchor box of the network for the i-th visible area, is a visible annotation box, is the intersection over union (IoU) function, N is the number of visible annotation boxes, and IoU() is the intersection over union (IoU) function;
[0147] The whole body region candidate box assignment label satisfies the following formula:
[0148]
[0149] in, is the i-th whole body region candidate box;
[0150] The candidate box assignment labels in the visible area satisfy the following formula:
[0151]
[0152] in, is the candidate box of the i-th visible area.
[0153] The calculation formula of the regression function of the anchor box in the whole body region extraction network is:
[0154]
[0155] The calculation formula of the regression loss function of the whole body detection frame obtained by regressing the whole body region candidate frame is:
[0156]
[0157] in, is the i-th whole body region candidate box assigned as a positive sample, Assign labels to the whole body annotation box, K1 is the number of positive samples in the whole body region extraction network stage, and M1 is the number of positive samples in the whole body region candidate box regression stage;
[0158] The calculation formula of the regression function of the anchor box in the visual area extraction network is:
[0159]
[0160] The regression loss function of the visible area candidate box after regression to obtain the visible detection box is calculated as:
[0161]
[0162] in, is the i-th visible area candidate box assigned as a positive sample, Assign labels to the visible annotation boxes, K2 is the number of positive samples in the visible area extraction network stage, and M2 is the number of positive samples in the visible area candidate box regression stage.
[0163] Step 3: Train the visibility network of the five parts of the human body according to the visible detection frame and the whole body detection frame obtained by the training of the training samples and the deep semantic features extracted by the backbone network, so as to enhance the visibility of the five parts of the human body, enhance the network's perception of occluded pedestrians, and calculate the enhanced loss function of the enhanced visibility of the five parts of the human body;
[0164] During the training process, for the whole body candidate box output by the whole body region extraction network, the whole body candidate box is divided into five parts according to the human body proportions, and five independent sub-networks are used to predict the visibility results of the five regions respectively. The dotted line in the figure represents that the sub-network is only used in training, and the branch is discarded in the inference stage so as not to increase the network inference cost.
[0165] The visibility network includes five independent sub-networks, each of which includes two fully connected layers connected in series. The visibility network training process is as follows:
[0166] The pedestrian's whole body is divided into five parts: head, left and right upper body, left and right lower body. The true visibility of each part (head, left and right upper body, left and right lower body) is calculated based on the whole body region candidate frame and visible frame annotation, specifically:
[0167]
[0168] Use a hard-coded form to convert the actual visibility to a visibility label:
[0169]
[0170] Where x represents the calculated true visibility result, and y represents the hard-coded converted visibility label;
[0171] Train a subnetwork for each body part and calculate the average loss for the five parts:
[0172]
[0173] Among them, y j It represents the visibility label of the jth part, p j Represents the visibility prediction result of the jth part. During the reasoning process, the visible whole body region extraction module branch is discarded to avoid increasing the reasoning cost.
[0174] Step 4: By optimizing the regression loss function and the enhancement loss function, their hyperparameters are adjusted until the total loss is minimized to obtain the detection model; the losses of the decoupling prediction module and the full-body frame visibility prediction module are calculated in parallel during the training process, and back-propagation is performed together to adjust the parameters of the overall model until the loss functions of the two modules are minimized, thereby completing the training of the detector.
[0175] Step 5, freeze the detection model, based on the visual detection frame and full body detection frame obtained by training the training samples, sample the corresponding features of the visual detection frame and full body detection frame that overlap with each other as training samples, train the similarity discrimination network and the recall strategy of suppression detection, so that it outputs the similarity of the features in the full body frame and the visual frame, and recalls the full body detection frame that should be retained but is suppressed due to the high overlap between pedestrians; the training of the suppression detection recall module only includes the training of the similarity discrimination network. In the process of training the similarity discrimination network, the entire detection model needs to be frozen to ensure the consistency of the output features.
[0176] The similarity discrimination network includes two fully connected layers connected in series, the number of intermediate channels of the two fully connected layers is 1024, and the similarity result is output through the Softmax function;
[0177] The training process of the similarity discrimination network and the recall strategy of suppression detection is as follows:
[0178] First, the detector is frozen, and the detector parameters are not updated during the training of the similarity discrimination network. Then, the visible region candidate box and the whole body region candidate box obtained by the visible region extraction network and the whole body region extraction network are output, and non-maximum suppression is performed on the visible region candidate box and the whole body region candidate box. The area intersection ratio (IOF) of the visible region candidate box and all the whole body region candidate boxes after non-maximum suppression is calculated. If the area intersection ratio (IOF) is greater than the threshold (0.6), it is considered to be a pair of samples.
[0179] Assign a positive label to each pair of examples:
[0180]
[0181] Among them, B v is the visual detection box result output by the detector, B f It is the full body box result output by the detector.
[0182] Except for the positive samples, all other samples are negative samples;
[0183] Based on the definition of positive and negative samples, the training samples are collected, and two memories are designed to store the features of the regions of interest corresponding to the positive and negative samples respectively, and the memories are continuously updated according to the first-in-first-out principle;
[0184] Based on the stored features, a fixed ratio of positive and negative samples are sampled each time for training. The label of the positive sample is 1 and the label of the negative sample is 0.
[0185] The loss function of the similarity discrimination network is:
[0186] L sim =LBCE (l pred ,l sim );
[0187] Among them, l pred is the similarity result output by the similarity discrimination network, l sim is the similarity label of positive and negative samples, L BCE Refers to the binary cross entropy loss function.
[0188] The decoupled prediction module guides the model to enhance its perception of the visible area of pedestrians. In addition, during the training process, the present invention utilizes the pedestrian full-body frame visibility prediction module to implicitly align the full-body region features with the visible region features, thereby improving the model's ability to distinguish the occluded parts of pedestrians. Finally, the present invention constructs a suppressed detection recall module, takes the visible frame and full-body frame detection results of the decoupled prediction module as input, uses the pedestrian's visible area features as a query, and calculates the similarity between the query and the full-body region features to judge and recall the suppressed full-body frame detection results. The method of the present invention can be combined with most proposal-based target detection frameworks to stably improve the detection performance of the detector.
[0189] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0190] It should be noted that the technical features not described in detail in the present invention can be implemented by any existing technology.
Claims
1. A pedestrian detection system for dense scenes, characterized in that: The system comprises: Backbone network, used to extract deep semantic features from images; The decoupled prediction module includes a pedestrian visible region extraction network and a full body region extraction network, which are used to obtain the pedestrian's visible region candidate frame and full body region candidate frame through the pedestrian's visible region extraction network and the full body region extraction network, respectively, and perform classification and regression calculations on them to obtain the full body detection frame, the full body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence; The full-body frame visibility prediction module divides the pedestrian's full-body frame into five parts according to the body parts, and enhances the network's perception of occluded pedestrians by predicting the visibility of the five parts; The suppressed detection recall module includes a similarity discrimination network and a suppressed detection recall strategy. The similarity discrimination network consists of three fully connected layers. The suppressed detection recall strategy is used to recall the full body frames that should have been retained but were suppressed due to the high overlap between pedestrians. The detection method is: (1) Input the image to be detected; (2) Obtain the deep semantic features of the image to be detected through the backbone network; (3) The deep semantic features are used to extract the visible region candidate boxes and the whole body region candidate boxes through the visible region extraction network and the whole body region extraction network; (4) The visible region candidate frame and the whole body region candidate frame are subjected to classification and regression calculation to obtain the whole body detection frame, the whole body detection frame classification confidence, the visible detection frame, and the visible detection frame classification confidence; (5) The full body detection frame, the full body detection frame classification confidence, the visible detection frame and the visible detection frame classification confidence suppress detection recall to obtain the image detection result.
2. The pedestrian detection system for dense scenes according to claim 1, characterized in that: The specific process of step (5) is as follows: (51) After processing multiple images, a full-body detection frame set B, a full-body detection frame classification confidence set S, a visible detection frame set H, and a visible detection frame classification confidence set J of the image to be detected are obtained, and the full-body detection frame set of the pedestrian to be recalled is set as R; (52) performing non-maximum suppression on the whole body detection frame set B to obtain a retained whole body detection frame set B1 and a suppressed whole body detection frame set B2, and arranging B1 and B2 in descending order according to classification confidence; (53) Perform non-maximum suppression on the visible detection box set H to obtain the retained visible detection box set H1, and arrange H1 in descending order according to classification confidence; (54) Take a visible box h from H1 in descending order according to the classification confidence j , calculate all full body boxes and h in B1 j The area intersection ratio IOF is calculated, and the whole body frame whose area intersection ratio IOF is greater than the threshold value 0.6 is taken out; (55) Extract the features of the corresponding area in the full-body frame whose area intersection ratio IOF is greater than the threshold 0.6 in step (54), and calculate the full-body frame and visible frame h obtained by similarity discrimination network j If the similarity of the corresponding regional features exceeds a certain threshold, h is deleted from H1 j ; (56) Take a visible box h from H1 in descending order according to the classification confidence j , repeat steps (54) and (55) until all visible frames in H1 have executed steps (54) and (55); (57) Take a visible box h from H1 after elimination in descending order according to the classification confidence j , calculate the visible frame h j The similarity between the features of the region corresponding to the full body frame in R, if the similarity exceeds the threshold, then repeat step (57); otherwise, proceed to step (58); (58) Take out the whole body detection box b in B2 in descending order of classification confidence i , calculate b i and h j IOF, if the foreground intersection value is less than the threshold 0.6, and b is calculated according to the similarity discrimination network i Corresponding regional features and h j If the similarity of the corresponding regional features is less than the threshold γ, then b i Put it into R and return to step (57); (59) The final set of full-body detection frames R of the pedestrians to be recalled and the set of full-body detection frames B1 retained after non-maximum suppression are merged to obtain the final image detection result.
3. The pedestrian detection system for dense scenes according to claim 2, characterized in that: All the full-body frames and visible frames h in the retained full-body detection frame set B1 in step (54) j The area intersection ratio IOF refers to the area of the whole body frame and the visible frame h in B1. j The area of the intersection divided by the visible box h j The specific formula is:
4. A method for constructing a pedestrian detection system for dense scenes, characterized in that: The construction method is used to construct the detection system according to any one of claims 1 to 3, and the construction method comprises: Step 1, obtaining a training sample set, the training sample set including a plurality of training samples, each of the training samples including an image containing a pedestrian and a corresponding visible frame annotation and a full body frame annotation; Step 2: extracting deep semantic features of training samples through the backbone network, and training the visible area extraction network and the whole body area extraction network through the deep semantic features so that they can extract visible area candidate frames and whole body area candidate frames; Then, classification and regression training is performed on the visible region candidate frame and the whole body region candidate frame, and the visible detection frame and the whole body detection frame are obtained through classification and regression, and the regression loss function of the visible region candidate frame and the whole body region candidate frame obtained through classification and regression is calculated; Step 3: Train the visibility network of the five parts of the human body according to the visible detection frame and the whole body detection frame obtained by the training of the training samples and the deep semantic features extracted by the backbone network, so as to enhance the visibility of the five parts of the human body, enhance the network's perception of occluded pedestrians, and calculate the enhanced loss function of the enhanced visibility of the five parts of the human body; Step 4: By optimizing the regression loss function and the enhancement loss function, their hyperparameters are adjusted until the total loss is minimized to obtain the detection model; Step 5: Freeze the detection model. Based on the visual detection frame and full-body detection frame obtained through training of the training samples, sample the corresponding features of the overlapping visual detection frame and full-body detection frame as training samples, train the similarity discrimination network and the recall strategy of suppression detection, so that it outputs the similarity of the features in the full-body frame and the visual frame, and recalls the full-body detection frame that should have been retained but was suppressed due to the high overlap between pedestrians.
5. The method for constructing a pedestrian detection system for dense scenes according to claim 4, characterized in that: In step 2, the process of training the visible area extraction network and the whole body area extraction network is as follows: Considering the anchor frames existing in the whole body region extraction network, calculating the intersection-and-union ratio of the current anchor frame and the whole body annotation frame in each training sample, and taking the whole body annotation frame with the largest intersection-and-union ratio with the current anchor frame as the assigned label of the current whole body region extraction network; Considering the visible annotation box corresponding to the whole body annotation box, determine whether the ratio of the area of the visible annotation box in the current anchor box to the entire visible annotation box reaches a threshold, and use whether the threshold is reached as the allocation standard of the current anchor box with respect to the visible annotation box corresponding to the whole body annotation box; If the current anchor frame has an assigned full-body annotation frame, and the current anchor frame meets the assignment criteria of the full-body annotation frame corresponding to the visible annotation frame, then the current anchor frame is assigned to the full-body annotation frame; Considering the anchor boxes existing in the visible area extraction network, the intersection-and-union ratio of the current anchor box and each visible annotation box is calculated, and the visible annotation box with the largest intersection-and-union ratio with the current anchor box is used as the intersection-and-union ratio label of the current anchor box; Calculate the intersection-and-union ratio of the current whole-body region candidate frame and each whole-body annotation frame, and use the whole-body annotation frame with the largest intersection-and-union ratio with the current whole-body region candidate frame as the intersection-and-union ratio assignment label of the current whole-body region candidate frame; at the same time, calculate the intersection-and-union ratio of the current visible region candidate frame and each visible annotation frame, and use the visible annotation frame with the largest intersection-and-union ratio with the current visible region candidate frame as the intersection-and-union ratio assignment label of the current visible region candidate frame; Iteratively train the visible region extraction network and the whole body region extraction network. After each training, calculate the regression loss of the whole body region candidate box and the regression loss of the visible region candidate box. Based on the regression loss function of the classification and regression of the visible area candidate boxes and the whole body area candidate boxes, the parameters of the backbone network, the visible area extraction network and the whole body area extraction network are adjusted.
6. The pedestrian detection system for dense scene and the construction method thereof according to claim 4, characterized in that: The anchor box assignment label in the whole body region extraction network satisfies the following formula: in, Extract the anchor box in the network for the i-th whole body region, Mark the whole body with a box. is the intersection over union (IoU) function, N is the number of full body annotation boxes, and IoU() is the intersection over union (IoU) function; The anchor frame in the whole body region extraction network must also satisfy the following formula: in, is the visible annotation box corresponding to the full-body annotation box with the largest IOU, θ is the given judgment threshold, For calculation exist The area in Function of area ratio; The anchor box assignment label in the visual area extraction network satisfies the following formula: in, Extract the anchor box of the network for the i-th visible area, is a visible annotation box, is the intersection over union (IoU) function, N is the number of visible annotation boxes, and IoU() is the intersection over union (IoU) function; The whole body region candidate box assignment label satisfies the following formula: in, is the i-th whole body region candidate box; The candidate box assignment labels in the visible area satisfy the following formula: in, is the candidate box of the i-th visible area.
7. The method for constructing a pedestrian detection system for dense scenes according to claim 4, characterized in that: The calculation formula of the regression function of the anchor box in the whole body region extraction network is: The calculation formula of the regression loss function of the whole body detection frame obtained by regressing the whole body region candidate frame is: in, is the i-th whole body region candidate box assigned as a positive sample, Assign labels to the whole body annotation box, K1 is the number of positive samples in the whole body region extraction network stage, and M1 is the number of positive samples in the whole body region candidate box regression stage; The calculation formula of the regression function of the anchor box in the visual area extraction network is: The regression loss function of the visible area candidate box after regression to obtain the visible detection box is calculated as: in, is the i-th visible area candidate box assigned as a positive sample, Assign labels to the visible annotation boxes, K2 is the number of positive samples in the visible area extraction network stage, and M2 is the number of positive samples in the visible area candidate box regression stage.
8. The method for constructing a pedestrian detection system for dense scenes according to claim 4, characterized in that: The visibility network includes five independent sub-networks, each of which includes two fully connected layers connected in series. The visibility network training process is as follows: The pedestrian's whole body is divided into five parts: head, left and right upper body, left and right lower body. The true visibility of each part is calculated based on the whole body region candidate frame and visible frame annotation, specifically: Use a hard-coded form to convert the actual visibility to a visibility label: Where x represents the calculated true visibility result, and y represents the hard-coded converted visibility label; Train a subnetwork for each body part and calculate the average loss for the five parts: Among them, y j It represents the visibility label of the jth part, p j Represents the visibility prediction result of the j-th part.
9. The method for constructing a pedestrian detection system for dense scenes according to claim 4, characterized in that: The similarity discrimination network includes two fully connected layers connected in series, the number of intermediate channels of the two fully connected layers is 1024, and the similarity result is output through the Softmax function; The training process of the similarity discrimination network and the recall strategy of suppression detection is as follows: First, the detector is frozen, and the detector parameters are not updated during the training of the similarity discrimination network. Then, the visible region candidate box and the whole body region candidate box obtained by the visible region extraction network and the whole body region extraction network are output, and non-maximum suppression is performed on the visible region candidate box and the whole body region candidate box. The area intersection ratio IOF of the visible region candidate box and all the whole body region candidate boxes after non-maximum suppression is calculated. If the area intersection ratio IOF is greater than the threshold, it is considered to be a pair of samples. Assign a positive label to each pair of examples: Among them, B v is the visual detection box result output by the detector, B f It is the full body box result output by the detector. Except for the positive samples, all other samples are negative samples; Based on the definition of positive and negative samples, the training samples are collected, and two memories are designed to store the features of the regions of interest corresponding to the positive and negative samples respectively, and the memories are continuously updated according to the first-in-first-out principle; Based on the stored features, a fixed ratio of positive and negative samples are sampled each time for training. The label of the positive sample is 1 and the label of the negative sample is 0.
10. The method for constructing a pedestrian detection system for dense scenes according to claim 9, characterized in that: The loss function of the similarity discrimination network is: L sim =L BCE (L pred ,L sim ); Among them, l pred is the similarity result output by the similarity discrimination network, l sim is the similarity label of positive and negative samples, L BCE Refers to the binary cross entropy loss function.