A pedestrian flow counting method integrating semantic information

By combining cross-modal learning of RMOT and ByteTrack and Kalman filter prediction and combining the trajectory correlation of Hungarian algorithm, the problems of low confidence target loss and trajectory fragmentation in pedestrian flow counting system are solved, achieving a more accurate and coherent Target tracking improves the system's multi-object and temporary state changes processing capabilities.

CN116977935BActive Publication Date: 2024-11-15BEIJING UNION UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310974997.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-04
Publication Date
2024-11-15
Estimated Expiration
2043-08-04

AI Technical Summary

Technical Problem

The existing pedestrian flow counting system is prone to losing low confidence targets in target detection, resulting in real target loss and trajectory fragmentation, making it difficult to effectively retain and filter background detection.

Method used

Using a method of fused semantic information, target tracking is carried out through RMOT and ByteTrack combined with all detection boxes, target position is predicted using cross-modal learning and Kalman filters, and trajectory correlation is performed in combination with Hungarian algorithms to ensure trajectory matching and background of low confidence targets. Information removal.

Benefits of technology

It effectively solves the problems of target loss and trajectory fragmentation, improves the accuracy and coherence of pedestrian detection and tracking, and enhances the system's multi-object and temporary state changes processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116977935B_ABST
    Figure CN116977935B_ABST
Patent Text Reader

Abstract

The present invention proposes a pedestrian flow counting method integrating semantic information, including: collecting pedestrian flow data information; performing target detection on the data information and recording target detection information; tracking the target through RMOT and ByteTrack based on the target detection information; performing threshold grouping on the target tracking results, and predicting the position of the target in the current frame based on the target tracking results to obtain the target track; performing track association on the grouped target tracking results and the target track, and counting based on the association results. The present invention can also perform track matching operations on the target detection frame when the target detection frame obtains low confidence, remove the real background information in the low confidence detection frame, and perform counting work according to the accurate tracking results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of machine vision and video image processing, and in particular relates to a pedestrian flow counting method integrating semantic information. Background Art

[0002] The pedestrian flow counting system is a system used to count the number of pedestrians and their movement trajectories, and is widely used in shopping malls, public transportation, tourist attractions, etc. The technologies required mainly include video image processing technology and deep learning technology.

[0003] Video image processing technology: The pedestrian flow counting system mainly obtains the image information of pedestrians through video surveillance cameras, and pre-processes and analyzes the images through video image processing technology, thereby realizing the functions of pedestrian detection, tracking and counting. The widely used technologies in visual algorithms include feature extraction, target detection, target tracking, motion estimation, etc.

[0004] Deep learning technology: In order to improve the accuracy of pedestrian detection and tracking, pedestrian flow counting systems use various deep learning technologies. Common machine learning algorithms include support vector machines, decision trees, random forests, etc.; while deep learning algorithms mainly use convolutional neural networks (CNN) and recurrent neural networks (RNN). These technologies can not only improve system performance, but also realize more functions, such as posture recognition, gender recognition, age recognition, etc.

[0005] Among them, multi-target tracking technology is an extremely critical link. This technology performs cascade matching based on the results of target detection and the feature similarity of the tracked targets, assigns a unique ID to each target and generates a target trajectory. The application of target tracking technology has many difficulties and challenges. Most traditional methods only assign IDs to detection frames with high confidence in target detection, and discard detection frames with confidence below the threshold, which may cause the real target to be lost and fragmented trajectories to be generated. How to retain low-confidence target detection frames to restore the real target and filter out background detections is an urgent problem that needs to be solved.

[0006] Therefore, the present invention proposes a pedestrian flow counting method based on ByteTrack that combines all detection boxes and fuses semantic information for target tracking. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a pedestrian flow counting method that integrates semantic information. When a target detection frame obtains low confidence, a trajectory matching operation is also performed on it, the real background information in the low-confidence detection frame is removed, and counting is performed based on accurate tracking results.

[0008] To achieve the above purpose, the present invention proposes a pedestrian flow counting method integrating semantic information, comprising:

[0009] Collect pedestrian flow data information;

[0010] Performing target detection on the data information and recording target detection information;

[0011] Based on the target detection information, target tracking is performed through RMOT and ByteTrack;

[0012] Threshold grouping is performed on the target tracking results, and the position of the target in the current frame is predicted based on the target tracking results to obtain the target track;

[0013] The grouped target tracking results are track-associated with the target track, and counting is performed based on the associated results.

[0014] Optionally, performing target detection on the data information includes:

[0015] Annotate the image dataset containing the target; the annotation content includes: target category, border position and target size;

[0016] Train the YOLOv6 model based on the labeled dataset;

[0017] Based on the trained YOLOv6 model, target detection is performed.

[0018] Optionally, recording the target detection information includes: recording target position information, target size information, and bounding box confidence;

[0019] Recording the target position information includes: recording the pixel coordinates, bounding box position and center point position of the target;

[0020] Recording the target size information includes: recording the width and height of the target;

[0021] Recording the bounding box confidence includes: assigning a bounding box to each detected object, and assigning a confidence score to each bounding box.

[0022] Optionally, target tracking via RMOT and ByteTrack includes:

[0023] RMOT is used for target tracking to obtain the position and motion state of the target. An early fusion module is used in RMOT to integrate the visual and language features before the deformable encoder layer. Given the feature map of the i-th layer Use 1×1 convolution to reduce its number of channels to d=128 and flatten it into a 2D tensor W iand H i are the width and height of the feature map of the i-th layer respectively; and the language features are projected onto the fully connected layer In , three independent fully connected layers transform visual and language features into Q, K, and V:

[0024]

[0025]

[0026]

[0027] Where W is the weight, P V and P L is the positional embedding of visual and linguistic features;

[0028] Do a matrix product on K and V, and use the resulting similarity matrix to weight the language features, i.e. d is the feature dimension, and then the original visual features are added to the language features to form a fusion feature

[0029]

[0030] A stack of deformable encoder layers is used after fusing the two modalities to promote cross-modal interaction:

[0031]

[0032] in Encoded as cross-modal embedding to facilitate reference prediction in subsequent decoders;

[0033] set up represents the decoder embedding of the t-1th frame, which is further converted into the trajectory query of the tth frame using the self-attention feedforward network, that is, Some of the N decoder embeddings correspond to empty or dropped targets, so we filter them out and keep only the N true embeddings to generate tracking queries based on their class scores make represents the detection query, which is randomly initialized for detecting new objects; these two queries are concatenated together and input into the decoder to learn the object representation D t :

[0034]

[0035] Among them, the number of output embeddings is N t , including tracking objects and detecting objects to obtain the position and motion state of the target;

[0036] The position and motion state of the target are input into the ByteTrack, and the ByteTrack algorithm is used to track the target in the video in real time to obtain the position and motion information of the target.

[0037] Optionally, threshold grouping of target tracking results includes:

[0038] Preset the confidence threshold τ;

[0039] Based on the bounding box confidence, all the detection boxes in each frame of the data information are divided. If the confidence of the detection box is greater than or equal to the threshold τ, the detection box is stored in the high-score detection box D high In the above example, if the confidence of the detection box is less than the threshold τ, the detection box is stored in the low-score detection box D low middle.

[0040] Optionally, predicting the position of the target in the current frame includes:

[0041] Initialization: At the beginning of tracking, a Kalman filter is initialized for each target track T obtained by target detection; wherein the Kalman filter includes a state vector, a state transfer matrix, an observation matrix, and a covariance matrix; wherein the state vector represents the state of the target in the current frame, including information such as position and speed; the state transfer matrix represents the evolution law of the target state; the observation matrix represents the observation value of the target in the current frame; and the covariance matrix represents the error covariance between the state vector and the observation value;

[0042] Prediction: The Kalman filter predicts the state of the target in the next frame based on the state vector and the state transfer matrix of the current frame, and estimates the covariance matrix of the predicted state;

[0043] Update: In the next frame, the position of the target is observed, and the observation value is represented by the observation matrix; the Kalman filter corrects the predicted state according to the observation matrix and the observation value to obtain the state estimate;

[0044] Loop: In the next frame, the state vector and the state transfer matrix after update are used to predict the state of the target in the next frame, and the above steps are repeated until the tracking is completed.

[0045] Optionally, associating the grouped target tracking results with the target track includes:

[0046] In the high-score detection frame D high Make the first association with all tracks T and keep the unmatched tracks;

[0047] In the low-score detection frame D low A second correlation is performed between the retained mismatched trajectories;

[0048] The trajectory association is completed based on the two association results.

[0049] Optionally, in the high score detection frame D high The first association between all trajectories T includes:

[0050] Detect the high-score detection frame D high The IoU between the predicted box and the trajectory T;

[0051] The IoU:

[0052]

[0053] Among them, area_inter represents the intersection of the predicted box and the real box, and area_union represents the union of the predicted box and the real box;

[0054] Use the Hungarian algorithm to complete the matching according to the IoU, and keep the unmatched detections in D remain In T remain Unmatched trajectories are retained in .

[0055] Optionally, in the low-score detection frame D low The second correlation between the retained unmatched trajectories includes:

[0056] Detect the low-score detection frame D low The IoU between the predicted boxes of the retained mismatched tracks;

[0057] Use the Hungarian algorithm to complete the matching according to the IoU, and keep the tracks that are still not matched to the detection box in T re-remain For T re-remain For each track in , when the track exists for more than the preset number of frames, it is deleted from the track, and the low-score detection box D that has not yet matched the track is deleted low .

[0058] Compared with the prior art, the present invention has the following advantages and technical effects:

[0059] 1. We address the limitations of existing reference understanding tasks by using RMOT for powerful cross-modal learning and provide an environment with multiple objects and temporary state changes.

[0060] 2. To solve the problem of non-negligible missing of real targets and fragmented tracks caused by retaining only high-scoring detection frames, ByteTrack is used to track by associating almost all detection frames, rather than just associating high-scoring detection frames, and using their similarity with tracklets to recover the real targets and filter out background detection frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0062] Figure 1 It is a schematic diagram of the overall process of an embodiment of the present invention;

[0063] Figure 2 This is a schematic diagram of the overall architecture of RMOT according to an embodiment of the present invention;

[0064] Figure 3 The figure is a schematic diagram showing the experimental results of an embodiment of the present invention. DETAILED DESCRIPTION

[0065] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0066] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] like Figure 1 As shown, this embodiment provides a method for counting pedestrian flow by integrating semantic information, including the following steps:

[0068] Step 1: Install fixed cameras to monitor the target area and collect data information.

[0069] Specifically, cameras are installed in areas where pedestrian flow counting is required, such as at the entrance of a pedestrian street. The installation of cameras needs to ensure that their monitoring range can completely cover the area where pedestrian flow counting is required, so as to ensure that complete video information can be collected and the number of pedestrians can be accurately counted.

[0070] Step 2: Train the YOLOv6 model to achieve target detection.

[0071] Specifically, the YOLOv6 algorithm is used to train the data set to solve the target detection task. This step requires the following sub-steps: data preparation, model training, model evaluation, and model application.

[0072] Step 2-1 Data preparation: First, you need to prepare an image dataset containing the target and annotate it. Annotation usually refers to marking the location and category of the target in the image. For the YOLOv6 model, the annotation usually uses the COCO format, which contains information such as the target category, bounding box location, and target size.

[0073] Step 2-2 Model training: The YOLOv6 model needs to be trained using the prepared dataset. The training process usually includes steps such as model initialization, forward propagation, back propagation, and gradient update. During the training process, it is necessary to select appropriate hyperparameters, loss functions, optimizers, etc. to improve the accuracy and generalization ability of the model. The batch size of the experiment in this invention is set to 32, and the initial learning rate is set to 10 -3 , the weight decay is 5×10 -4 , the optimizer uses SGD. The overall loss function is:

[0074]

[0075] Among them, L cls is the weighted focal loss for classification, L1 and generalized IoU are the losses L iou For bounding box regression, λ iou With λ L1 is the regularization parameter. The training process needs to be performed on the GPU to improve the training speed and efficiency.

[0076] Step 2-3 Model evaluation: After training is completed, the performance of the model needs to be evaluated. Generally, accuracy (Acc), precision (Precision), recall (Recall), mean average precision (mAP), etc. can be used. The evaluation can be performed through cross-validation, test set validation, etc. The evaluation results can be used to adjust model parameters, improve training data, etc., to improve the performance and generalization ability of the model. The main evaluation indicators used in the target tracking process of the experiment of the present invention are Acc, Precision and Reacll.

[0077] Step 2-4 Model application: Apply the trained model to downstream tasks. In the present invention, the downstream tasks are multi-target tracking and pedestrian flow counting.

[0078] Step three: record target detection information.

[0079] Specifically, the input image and output results of the target detection task are recorded for subsequent analysis and application. This step requires recording the basic information of the image and the information of the detection results. The information recorded in the experiment of the present invention mainly includes the following aspects:

[0080] 1. Target location information: Record the precise location information of the target in the video by recording the pixel coordinates, bounding box position, center point position, etc. of the target;

[0081] 2. Target size information: record the size of the target, by recording the width, height, etc. of the target;

[0082] 3. Bounding box confidence: For each detected target, a bounding box is assigned to it, and a confidence score is assigned to each bounding box. This confidence score is obtained through the output of the target detection algorithm (the algorithm used in the experiment of this invention is YOLOv6).

[0083] Step 4: Target tracking through RMOT and ByteTrack.

[0084] This step combines RMOT and ByteTrack. By combining RMOT and ByteTrack, we can make full use of their respective advantages to achieve more accurate and efficient target tracking. The core idea of ​​RMOT is to use language expressions as semantic clues to guide the prediction of multi-object tracking. It can solve the problem that previous tasks can only detect objects pointed to by a single text, making the task closer to the real environment. At the same time, it can also handle multi-target scenes and predict the positions of any number of reference targets in the video. ByteTrack is a tracking method based on the tracking-by-detection paradigm. By using the similarity between the detection box and the tracking trajectory, while retaining the high-score detection results, the background is removed from the low-score detection results, and the real objects (difficult samples such as occlusion and blur) are excavated, thereby reducing missed detections and improving the coherence of the trajectory.

[0085] First, RMOT is used to track the target, and information such as the target's position and motion state can be obtained. Specifically, RMOT can determine the target's position and motion state by comparing the distance and direction between the target and the reference point. Then, the target position and state information obtained by RMOT can be used as input to perform target detection and tracking using ByteTrack. By using the target position and state information obtained by RMOT, ByteTrack can detect and track the target more quickly and accurately, and the new tracking results can be continuously passed to RMOT for online instantiation and update.

[0086] Specifically, Referring Multi-Object Tracking (RMOT) can integrate semantic information to accurately track specific targets. Figure 2As shown in the figure, the ByteTrack algorithm combines deep learning and traditional image processing technology in the tracking process, and has strong robustness and real-time performance.

[0087] Step 4-1: RMOT uses an early fusion module to integrate visual and language features before the deformable encoder layer. Given the feature map of the i-th layer Use 1×1 convolution to reduce its number of channels to d=128 and flatten it into a 2D tensor W i and H i are the width and height of the feature map of the i-th layer respectively. To retain the same channels as the visual features, a fully connected layer is used to project the language features to In , three independent fully connected layers transform visual and language features into Q, K, and V:

[0088]

[0089]

[0090]

[0091] Where W is the weight, P V and P L It is the position embedding of visual and language features.

[0092] Step 4-2: Perform matrix product on K and V, and use the generated similarity matrix to weight the language features, that is, d is the feature dimension, and then the original visual features are added to the language features to form a fusion feature

[0093]

[0094] Step 4-3: Use a stack of deformable encoder layers after fusing the two modalities to promote cross-modal interaction:

[0095]

[0096] in Encoded as cross-modal embedding, it facilitates reference prediction in subsequent decoders.

[0097] Step 4-4: Set represents the decoder embedding of the t-1th frame, which is further converted into the trajectory query of the tth frame using the self-attention feedforward network (FFN), that is, Some of the N decoder embeddings correspond to empty or dropped targets, so we filter them out and keep only the N true embeddings to generate tracking queries based on their class scores make represents the detection query, which is randomly initialized for detecting new objects. In practice, these two queries are concatenated and input to the decoder to learn the object representation D t :

[0098]

[0099] Among them, the number of output embeddings is N t , including tracking objects and detecting objects.

[0100] Step 4-5: Use the ByteTrack algorithm to track the target in the video in real time to obtain the target's position and motion information. Based on the video stream provided by the user as input, the target in the video is continuously tracked. After the tracking is completed, the user can obtain the target's position, size, motion trajectory and other information, as well as analyze and predict the target's state.

[0101] Step 5: Group according to whether the confidence of the detection box is greater than the threshold.

[0102] Specifically, first, a confidence threshold τ needs to be set. In the present invention, the confidence is set to τ = 0.5. According to the confidence information obtained in step 3, all the detection frames of each frame in the video are divided, that is, the detection frames with confidence greater than or equal to the threshold τ are stored in the high-score detection frame D high In the above example, if the confidence of the detection box is less than the threshold τ, the detection box is stored in the low-score detection box D low This step can help us screen and classify the target detection results, thereby improving the accuracy and efficiency of detection.

[0103] Step 6: Predict the new position of each track T in the current frame through the Kalman filter: The Kalman filter algorithm is a filtering algorithm based on the state space model, which has high prediction accuracy and robustness and can be applied to a variety of different control and prediction problems. Use the Kalman filter algorithm to predict the position of the target in the current frame. The input of this step includes the position and velocity information of the target in the previous frame, as well as the observation data of the current frame. The Kalman filter will use the above information to predict the position and velocity of the target in the current frame, and give the predicted value and error range.

[0104] Specifically, the process of predicting the new position of each track T in the current frame through the Kalman filter is as follows:

[0105] 1. Initialization: At the beginning of tracking, the Kalman filter is initialized for each target track T. The Kalman filter includes components such as state vector, state transfer matrix, observation matrix, and covariance matrix. Among them, the state vector represents the state of the target in the current frame, including position, speed and other information; the state transfer matrix represents the evolution law of the target state; the observation matrix represents the observation value of the target in the current frame; the covariance matrix represents the error covariance between the state vector and the observation value; among them, the target track T is obtained by the target detection algorithm in step 2. The target detection algorithm will detect areas that may be targets in the video frame, and then infer the position, size and other information of each target in the current frame based on these areas.

[0106] 2. Prediction: During the tracking process, the Kalman filter predicts the state of the target in the next frame based on the state vector and state transfer matrix. Specifically, the Kalman filter predicts the state of the target in the next frame based on the state vector and state transfer matrix of the current frame, and estimates the covariance matrix of the predicted state;

[0107] 3. Update: In the next frame, the position of the target will be observed, and this observation value can be represented by the observation matrix. The Kalman filter will correct the predicted state based on the observation matrix and the observation value to obtain a more accurate state estimate;

[0108] 4. Loop: In the next frame, use the updated state vector and state transfer matrix to predict the state of the target in the next frame, and repeat the above steps until the tracking is completed.

[0109] Predicting the new position of each track T in the current frame through the Kalman filter can help us achieve target tracking.

[0110] Step 7: Use the Hungarian algorithm to match, save and display the tracking results: Use the Hungarian algorithm to match the trajectories of multiple targets, and save and display the matching results. This step requires the input of trajectory data of multiple targets, and obtains a unique identifier for each target by matching the trajectories using the Hungarian algorithm. The Hungarian algorithm is an algorithm based on graph theory that can perform optimal matching between multiple targets with high matching accuracy and efficiency. After the matching is completed, the matching results can be saved to a storage medium such as a database for subsequent analysis and application. At the same time, the tracking results are displayed through visualization tools for manual inspection and verification.

[0111] Step 8: Associating the detection frame with the trajectory: According to steps 1 to 7, the association method of the present invention can be divided into two stages for trajectory association.

[0112] Step 8-1: First association in high score detection box D high and all tracks T (including the missing track Tlost ) between the two. high The IoU between the predicted box and the trajectory T is calculated as follows:

[0113] Among them, area_inter represents the intersection of the predicted box and the true box, and area_union represents the union of the predicted box and the true box.

[0114] Then the Hungarian algorithm mentioned in step 7 is used to complete the matching based on the similarity, and the unmatched detections are retained in D remain In T remain Unmatched trajectories are retained in .

[0115] Among them, using the Hungarian algorithm to complete matching based on similarity includes:

[0116] After calculating the IoU between two object boxes in object detection and representing them as a similarity matrix, it can be used as input to the Hungarian algorithm to find the best match.

[0117] The core of the Hungarian algorithm is to generate as many matching pairs of targets and trajectories as possible. The following are the steps to complete object box matching in target detection using the Hungarian algorithm:

[0118] (1) Sort all object frames by similarity from high to low;

[0119] (2) Select the object frame with the highest similarity from the similarity matrix and match it with its corresponding target;

[0120] (3) Delete the matched targets and object frames from the matrix, and subtract the minimum values ​​of the corresponding rows and columns from the similarities of the remaining unmatched targets and object frames;

[0121] (4) Repeat (2) and (3) until all object boxes are matched.

[0122] Ultimately, the Hungarian algorithm returns a set of matched object detection boxes. If there are unmatched objects or object boxes, they can be considered as new objects or object boxes, or ignored.

[0123] Step 8-2: The second association is in the low-score detection box D low and the remaining trajectory T after step 8-1 remain The tracks that are still not matched to the detection box are kept in T re-remain For T re-remainFor each track in , it is deleted from the track only when it exists for more than a certain number of frames (set to 40 frames in this embodiment). Delete the low-scoring detection frame that has not yet matched the track, because it is regarded as the background and does not contain any target information. Because the low-scoring detection frame usually contains severe occlusion or motion blur, the appearance features are unreliable, so it is important to use IoU alone as the similarity in this step.

[0124] Step 8-3: Evaluate the tracking results by calculating values ​​such as MOTA and IDF1. The calculation method is as follows:

[0125]

[0126]

[0127] Among them, FN is False Negative, the sum of the number of missed alarms in the entire video; FP is False Positive, the sum of the number of false alarms in the entire video; IDSW is ID Switch (total number of ID switches, mismatches), the sum of the number of mismatches in the entire video; GT is the number of Ground Truth objects, the sum of the number of GTs in the entire video. IDTP, IDFP and IDFN are the number of true IDs, false positive IDs and false negative IDs respectively.

[0128] After steps 1-8, this example realizes the target tracking function. Step 9 sets the counting area and the corresponding counting rules in the video area, obtains the motion information (motion trajectory and direction, etc.) of the pedestrian according to the target tracking results in each frame, and thus determines whether the pedestrian passes through the counting area, and then increases the value in the corresponding counting column according to the direction of the pedestrian.

[0129] Step 9: Set the counting area and counting rules and count the traffic:

[0130] Specifically, a suitable area is selected as the counting area according to the actual situation. The counting area selected in the experiment of the present invention is at the entrance of the pedestrian street, and the counting area is set to a rectangular shape. Figure 3 As shown, two range lines are set within the range of video surveillance. After setting the counting area, it is necessary to further set the counting rules. The counting rules usually include the direction of entry and exit and the counting method. The counting rules of this embodiment are set as follows: if the pedestrian passes through the light-colored area first and then the white area, the number of people entering (or going up) will be increased by one, otherwise the number of people exiting (or going down) will be increased by one. After setting the counting area and counting rules, the image processing algorithm mentioned in the above steps is used to detect and track people or vehicles in the counting area, and then the traffic statistics are performed according to the counting rules, and the results are dynamically displayed in the upper right corner of the video so as to grasp the personnel flow in the current area in real time.

[0131] The innovation of this embodiment is:

[0132] Target tracking is performed through ByteTrack and fusion of semantic information to obtain accurate tracking results for subsequent pedestrian counting;

[0133] Replace ByteTrack's detector with the YOLOv6 network with stronger representation capabilities, faster detection speed, and higher AP accuracy;

[0134] The backbone of the neural network used to fuse semantic information was changed from ResNet50 to DenseNet. DenseNet has the same basic idea as ResNet, but it establishes dense connections between all previous layers and the following layers, and achieves feature reuse by connecting features on channels. These features enable DenseNet to achieve better performance than ResNet with fewer parameters and lower computational costs.

[0135] Experiments have shown that the present embodiment achieves a MOTA value of 0.764 and an IDF1 value of 0.806. Pedestrian flow is counted in the target video by manual counting and system counting. By comparison, it is found that the present invention achieves the same flow result as manual counting, which proves that the method of the present invention can replace the manual counting method and improve the efficiency of pedestrian counting, so as to help managers respond to different pedestrian flow situations more promptly.

[0136] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A pedestrian flow counting method integrating semantic information, It is characterized in that include: Collect pedestrian flow data information; Performing target detection on the data information and recording target detection information; Recording the target detection information includes: recording target position information, target size information, and bounding box confidence; Recording the target position information includes: recording the pixel coordinates, bounding box position and center point position of the target; Recording the target size information includes: recording the width and height of the target; Recording the bounding box confidence includes: assigning a bounding box to each detected object and assigning a confidence score to each bounding box; Based on the target detection information, target tracking is performed through RMOT and ByteTrack; Target tracking via RMOT and ByteTrack includes: RMOT is used for target tracking to obtain the position and motion state of the target. An early fusion module is used in RMOT to integrate the visual and language features before the deformable encoder layer. Given the feature map of the i-th layer Use 1×1 convolution to reduce its number of channels to d=128 and flatten it into a 2D tensor W i and H i are the width and height of the feature map of the i-th layer respectively; and the language features are projected onto the fully connected layer In , three independent fully connected layers transform visual and language features into Q, K and V, which are expressed as follows: Where W is the weight, P V and P L is the positional embedding of visual and linguistic features; Do a matrix product on K and V, and use the resulting similarity matrix to weight the language features, i.e. d is the feature dimension, and then the original visual features are added to the language features to form a fusion feature A stack of deformable encoder layers is used after fusing the two modalities to promote cross-modal interaction: in Encoded as cross-modal embedding to facilitate reference prediction in subsequent decoders; set up represents the decoder embedding of the t-1th frame, which is further converted into the trajectory query of the tth frame using the self-attention feedforward network, that is, Some of the N decoder embeddings correspond to empty or dropped targets, so we filter them out and keep only the N true embeddings to generate tracking queries based on their class scores make represents the detection query, which is randomly initialized for detecting new objects; these two queries are concatenated together and input into the decoder to learn the object representation D t : Among them, the number of output embeddings is N t , including tracking objects and detecting objects to obtain the position and motion state of the target; The position and motion state of the target are input into the ByteTrack, and the target in the video is tracked in real time using the ByteTrack algorithm to obtain the position and motion information of the target; Threshold grouping is performed on the target tracking results, and the position of the target in the current frame is predicted based on the target tracking results to obtain the target track; Threshold grouping of target tracking results includes: Preset the confidence threshold τ; Based on the bounding box confidence, all the detection boxes in each frame of the data information are divided. If the confidence of the detection box is greater than or equal to the threshold τ, the detection box is stored in the high-score detection box D high In the above example, if the confidence of the detection box is less than the threshold τ, the detection box is stored in the low-score detection box D low middle; Associating the grouped target tracking results with the target tracks, and counting based on the association results; Associating the grouped target tracking results with the target track includes: In the high-score detection frame D high Make the first association with all tracks T and keep the unmatched tracks; In the low-score detection frame D low A second correlation is performed between the retained mismatched trajectories; completing the trajectory association based on the two association results; In the high-score detection frame D high The first association between all tracks T includes: Detect the high-score detection frame D high The IoU between the predicted box and the trajectory T; The IoU: Among them, area_inter represents the intersection of the predicted box and the real box, and area_union represents the union of the predicted box and the real box; Use the Hungarian algorithm to complete the matching according to the IoU, and keep the unmatched detections in D remain In T remain Keep the unmatched trajectories in In the low-score detection frame D low The second correlation with the retained unmatched trajectories includes: Detect the low-score detection frame D low The IoU between the predicted boxes of the retained mismatched tracks; Use the Hungarian algorithm to complete the matching according to the IoU, and keep the tracks that are still not matched to the detection box in T re-remain For T re-remain For each track in , when the track exists for more than the preset number of frames, it is deleted from the track, and the low-score detection box D that has not yet matched the track is deleted low .

2. According to the method for counting pedestrian flow by integrating semantic information in claim 1, It is characterized in that Performing target detection on the data information includes: Annotate the image dataset containing the target; the annotation content includes: target category, border position and target size; Train the YOLOv6 model based on the labeled dataset; Based on the trained YOLOv6 model, target detection is performed.

3. The pedestrian flow counting method integrating semantic information according to claim 1, It is characterized in that Predicting the position of the target in the current frame includes: Initialization: At the beginning of tracking, a Kalman filter is initialized for each target track T obtained by target detection; wherein the Kalman filter includes a state vector, a state transfer matrix, an observation matrix, and a covariance matrix; wherein the state vector represents the state of the target in the current frame, including position and velocity information; the state transfer matrix represents the evolution law of the target state; the observation matrix represents the observation value of the target in the current frame; and the covariance matrix represents the error covariance between the state vector and the observation value; Prediction: The Kalman filter predicts the state of the target in the next frame based on the state vector and the state transfer matrix of the current frame, and estimates the covariance matrix of the predicted state; Update: In the next frame, the position of the target is observed, and the observation value is represented by the observation matrix; the Kalman filter corrects the predicted state according to the observation matrix and the observation value to obtain the state estimate; Loop: In the next frame, the state vector and the state transfer matrix after update are used to predict the state of the target in the next frame, and the initialization step, the prediction step, and the update step are repeated until the tracking is completed.