Library people flow monitoring method and system based on multi-mode and trajectory fusion

Abstract

The invention discloses a library people flow monitoring method and system based on multi-mode and trajectory fusion. The method comprises the steps of collecting a video stream, generating a weight image by using an attention mechanism to enhance a pedestrian area, extracting a pedestrian frame through YOLOv11 target detection, and positioning a pedestrian subject and separating a background in combination with a segmentation model; feature points and descriptors are extracted through SuperPoint, inter-frame motion parameters are calculated through feature matching, and tracks are predicted and verified by means of Kalman filtering; and finally, associating pedestrian tracks through unique IDs, and dynamically tracking in and out states to count the pedestrian flow. According to the method, the problems of shielding, small target missing detection and track drifting are effectively solved, and high-precision people flow statistics is realized; more accurate and robust pedestrian detection and tracking are realized through multi-modal data fusion and dynamic trajectory fusion; through the schemes of feature point extraction, Kalman filtering and the like, the prediction of pedestrian motion trails is optimized, and the real-time performance and precision of pedestrian flow monitoring are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking flow statistics, and in particular to a library passenger flow monitoring method and system based on multimodality and trajectory fusion. Background Art

[0002] Libraries, as crowded places, face overloaded traffic, a primary safety management challenge. Real-time monitoring of traffic flow not only allows for a dynamic understanding of the distribution of people within the library, assisting staff in optimizing seating and resource allocation, but also enables intelligent early warning mechanisms to trigger intervention measures when crowd density approaches safety thresholds, effectively preventing stampedes.

[0003] Current mainstream pedestrian flow monitoring technology faces multiple technical bottlenecks: First, during peak hours, the increased occlusion caused by dense crowds, the increased randomness of pedestrian trajectories, and the aggregation of small target groups can significantly reduce the consistency between pedestrian counts and cross-frame target tracking, and may even lead to tracking failure. Second, the traditional single-frame detection framework has inherent flaws—the detection frame covers both the pedestrian target itself and dynamic environmental elements. Since pedestrians are in a state of continuous motion, the dynamic changes in background features within the target frame interfere with motion trajectory modeling, resulting in trajectory prediction deviations (inaccurate calculated motion trajectories). Third, for small and low-featured targets (such as children, short pedestrians, or individuals obstructing the view from the side), feature sparsity can easily lead to missed detections, resulting in inaccurate crowd flow monitoring. These technical shortcomings jointly restrict the accuracy and stability of library crowd flow monitoring. Summary of the Invention

[0004] The embodiment of the present invention provides a library traffic flow monitoring method and system based on multimodality and trajectory fusion, so as to solve the technical problems of low monitoring and tracking speed and low accuracy of existing library traffic flow monitoring solutions.

[0005] A library traffic monitoring method based on multimodal and trajectory fusion includes:

[0006] Collect the video stream of the library and pre-process it to obtain the image to be detected containing weight information of the area to be detected;

[0007] Perform target detection on the image to be detected containing weight information and obtain the current pedestrian detection frame of the library;

[0008] Based on the position of the pedestrian detection frame, the original image is segmented using the segmentation model to obtain the segmented image area;

[0009] Preprocess the segmented image area to extract the image information corresponding to the pedestrians, and extract and describe the feature points of multiple pedestrians in the same image, and store them as corresponding feature point information respectively;

[0010] Based on the feature point information, the correspondence between pedestrians in action is established, and the inter-frame motion results are determined through the correspondence;

[0011] Based on the inter-frame motion results, the Kalman filter is used to predict the trajectory, and the feature point extraction and corresponding matching relationship are combined for verification to obtain the motion information of the same pedestrian in consecutive frames;

[0012] Based on the motion information and association relationship, the unique detection information of pedestrians is determined and assigned a unique ID for storage. The unique ID is used to track the entry and exit of each pedestrian to count the flow of people in the library.

[0013] A library traffic monitoring system based on multimodal and trajectory fusion, including:

[0014] The acquisition module is used to acquire the video stream of the library and perform preprocessing to obtain the image to be detected containing weight information of the area to be detected;

[0015] The target detection module is used to perform target detection on the image to be detected containing weight information and obtain the current pedestrian detection frame of the library;

[0016] The region segmentation module is used to segment the original image based on the position of the pedestrian detection frame using the segmentation model to obtain the segmented image area;

[0017] The feature extraction module is used to pre-process the segmented image area, extract the image information corresponding to the pedestrian, and extract and describe the feature points of multiple pedestrians in the same image, and store them as corresponding feature point information respectively;

[0018] The data association module is used to establish the correspondence between pedestrians in action based on feature point information and determine the inter-frame motion results through the correspondence;

[0019] The trajectory tracking module is used to predict the trajectory based on the inter-frame motion results using Kalman filtering, and verify it by combining feature point extraction and corresponding matching relationships to obtain the motion information of the same pedestrian in consecutive frames;

[0020] The traffic statistics module is used to determine the unique detection information of pedestrians based on movement information and association relationships, and assign a unique ID for storage. The unique ID is used to track the entry and exit of each pedestrian to count the flow of people in the library.

[0021] The present invention has the following beneficial effects:

[0022] This invention achieves more accurate and robust pedestrian detection and tracking through multimodal data fusion and dynamic trajectory fusion technology. It dynamically enhances pedestrian regional features through attention mechanism weight allocation, and combines the segmentation model to separate the pedestrian body from the dynamic background, significantly reducing background noise interference. At the same time, based on SuperPoint feature point matching and Kalman filter trajectory prediction, trajectory continuity is restored through multi-frame feature association in occluded scenes, which optimises the prediction of pedestrian motion trajectories, thereby improving the real-time and accuracy of pedestrian flow monitoring. It significantly reduces trajectory prediction deviations caused by dynamic changes in background features within the target frame, and also makes up for the shortcomings of traditional single-frame detection frameworks, improving the ability to identify small targets or pedestrians with unclear features, thereby reducing missed detections and enhancing the overall accuracy of pedestrian flow monitoring.

[0023] This invention uses the YOLOv11 lightweight model combined with private dataset training to optimize detection capabilities for scenarios such as low pedestrians and side occlusions, significantly improving the recall rate of small targets; the main pedestrian area is extracted through mask segmentation technology, avoiding background feature interference within the detection frame, and significantly improving the accuracy of feature point matching.

[0024] Through multi-dimensional state modeling (motion parameters + feature descriptors) and a unique ID binding mechanism, the present invention can maintain trajectory consistency even in the presence of short-term pedestrian occlusion, rapid movement, or densely populated scenes. Compared with traditional methods that suffer from tracking interruptions due to background interference or feature loss, the present invention significantly reduces the risk of ID confusion through the collaborative optimization of historical trajectory prediction and real-time feature matching. In practical applications, the density of pedestrian traffic in hot spots (such as entrances and reading areas) can be counted in real time, dynamically triggering passenger flow diversion plans, and providing data support for library resource scheduling (such as seat allocation and self-service equipment optimization), helping to improve the management efficiency of smart libraries. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 This is a flow chart of a library passenger flow monitoring method based on multimodality and trajectory fusion in one embodiment of the present invention;

[0027] Figure 2 1 is a schematic diagram of a library passenger flow monitoring system based on multimodality and trajectory fusion in one embodiment of the present invention;

[0028] Figure 3is another schematic diagram of a library passenger flow monitoring system based on multimodality and trajectory fusion in one embodiment of the present invention;

[0029] Figure 4 FIG. 1 is a schematic diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The library traffic flow monitoring method based on multimodal and trajectory fusion provided by the embodiment of the present invention can be applied as follows: Figure 3 and Figure 4 Specifically, the library passenger flow monitoring method based on multimodal and trajectory fusion is applied in the library passenger flow monitoring system based on multimodal and trajectory fusion, and the library passenger flow monitoring system based on multimodal and trajectory fusion includes the following: Figure 3 The client and server shown communicate over a network to improve the speed and accuracy of library visitor flow monitoring and tracking. The client, also known as the user end, is the program that interfaces with the server and provides local services to clients. The client can be installed on, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0032] In one embodiment, if Figure 1 As shown in the figure, a library traffic monitoring method based on multimodal and trajectory fusion is provided. Figure 3 The server in the example is used as an example, and the steps are as follows:

[0033] S1, collect the video stream of the library and pre-process it to obtain the image to be detected containing weight information of the area to be detected;

[0034] S2. Perform target detection on the image to be detected containing weight information to obtain the current pedestrian detection frame of the library;

[0035] S3. Based on the position of the pedestrian detection frame, the original image is segmented using the segmentation model to obtain a segmented image area;

[0036] S4. Preprocess the segmented image area to extract image information corresponding to the pedestrians, and extract and describe feature points for multiple pedestrians in the same image, and store them as corresponding feature point information respectively;

[0037] S5. Based on the feature point information, establish a correspondence relationship between pedestrians in action, and determine the inter-frame motion result through the correspondence relationship;

[0038] S6. Based on the inter-frame motion results, the trajectory is predicted using Kalman filtering, and verified by combining feature point extraction and corresponding matching relationships to obtain the motion information of the same pedestrian in consecutive frames;

[0039] S7. Based on the motion information and association relationship, the unique detection information of the pedestrian is determined and assigned a unique ID for storage. The unique ID is used to track the entry and exit of each pedestrian to count the flow of people in the library.

[0040] In one embodiment, step S1: capturing a video stream from the library and preprocessing it to obtain an image of the area to be detected containing weight information. It is understood that by capturing the video stream within the library, scene information within the library can be obtained in real time, preparing high-quality image data for subsequent target detection and analysis. This step includes the following sub-steps:

[0041] S101 , using a front-end camera to collect a video stream of the area to be detected, extracting image frames from the video stream, and performing grayscale, equalization, and denoising processing on the image frames to obtain an image F to be detected.

[0042] S102: performing weight assignment on the image to be detected F based on the attention mechanism to obtain weight information of the area to be detected; wherein the expression of the weight assignment method is:

[0043] A=σ(Conv 1×1 ([F avg ; F max ])) (1)

[0044] Where A∈R H×W represents the weight distribution of the associated image spatial information, H×W is the size information of the image, F avg and F max are channel dimension average pooling and maximum pooling respectively, and σ is the Sigmoid activation function.

[0045] As can be understood, after preprocessing the image frames in the video stream, the resulting image F to be detected has greater clarity and contrast, providing higher-quality input data for subsequent target detection. By focusing on pedestrian areas through the attention mechanism, the missed detection rate is significantly reduced in scenes with dense crowds or complex backgrounds. As pedestrians move, the weight map dynamically adjusts the focus area, providing more accurate initial information for subsequent segmentation, feature extraction, and trajectory prediction, significantly reducing tracking failures caused by target overlap or deformation.

[0046] In one embodiment, step S2: performing target detection on the image to be detected containing weight information to obtain the current pedestrian detection frame of the library. This step is understandably intended to address issues of missed detection and false detection caused by pedestrian occlusion, background interference, and sparse small target features in traditional single-frame detection. It also includes the following sub-steps:

[0047] S201. Obtain a pedestrian dataset based on public and private datasets. Public datasets include Caltech Pedestrian and KITTI, while private datasets are annotated for special scenarios such as low-height pedestrians, side occlusions, and dense crowds in libraries. This allows for the construction of a pedestrian dataset covering diverse scenarios, enhancing the model's adaptability to complex environments. This can reduce detection bias caused by a single scenario and avoid missed detection of small objects (such as children and sideways pedestrians) and dense crowds.

[0048] S202. Based on the pedestrian dataset, the training, test, and validation sets are weighted at an 8:1:1 ratio. The model is trained using the YOLOv11 algorithm to obtain the optimal training model weights. As can be seen, model training achieves high recall (reducing missed detections) and high precision (reducing false detections) in library scenarios, and accurately locates pedestrians even in densely populated or low-profile pedestrian scenes.

[0049] S203: Based on the optimal training model weights, the model is deployed to the terminal to perform pedestrian detection, and a detection result frame containing all pedestrian targets is output. Duplicate frames are sorted based on the non-maximum suppression model, and redundant frames are deleted to obtain pedestrian detection frames. The mathematical expression of the non-maximum suppression model is:

[0050]

[0051] Among them, β represents the set of all detected bounding boxes, b i and b j Represents two different bounding boxes, Area(b i ∩b j ) means calculating the intersection area of ​​two bounding boxes, Area(b i ∪b j) represents the calculation of the union area of ​​two bounding boxes, θ NMS Represents the set threshold parameter; when the intersection over union (IoU) of two bounding boxes is less than the threshold θ NMS When , it is confirmed that different targets are detected and will not suppress each other, and the two bounding boxes are retained. Understandably, this step eliminates duplicate detection boxes, reduces the computational complexity of subsequent tracking, improves the neatness of the detection results, and avoids the interference of track association due to redundant boxes.

[0052] In one embodiment, step S3: based on the position of the pedestrian detection frame, using a segmentation model to segment the original image to obtain a segmented image region. It can be understood that through refined segmentation and dynamic background suppression, the core problem of the pedestrian detection frame in the library scene being susceptible to background interference is solved. Stable segmentation performance can still be maintained in scenes with dense crowds and severe occlusion of small targets. This includes the following sub-steps:

[0053] S301: Obtain the location of the pedestrian detection frame and, based on the location of the pedestrian detection frame, extract the corresponding partial image from the original image. This avoids segmenting the entire image, improving the efficiency of the subsequent segmentation model. It also eliminates background interference outside the detection frame (such as bookshelves, chairs, and other pedestrians), providing clean data input for refined segmentation.

[0054] S302: Input the local image into a pre-trained segmentation model for feature segmentation, extract the inter-frame features of the target pedestrian, and dynamically generate mask information; understandably, when the crowd is dense or the pedestrian is partially blocked, the pedestrian outline can still be accurately outlined; the background noise in the detection frame is filtered by the mask, thereby improving the accuracy of subsequent feature extraction.

[0055] S303: Using the mask information, a segmented image region of the pedestrian target is extracted from the original image. The segmented image region includes the target region and the background region. Based on the mask information, the number of all pedestrians in the single-frame image is counted. This segmented pedestrian region provides high-quality input for subsequent feature point extraction, reducing feature matching errors caused by background feature contamination.

[0056] In this way, through steps S302-S303, the main body of pedestrians can be accurately separated in complex scenes (such as dense crowds and pedestrian occlusions), avoiding the influence of background interference on trajectory prediction; when the bodies of two pedestrians partially overlap, the segmentation model can distinguish the contours of the two and prevent trajectory confusion. The segmentation model learns the background features of the library scene through training. During the segmentation process, it automatically ignores the static background and only retains the dynamic area of ​​the pedestrian. Even if the position of the detection frame changes due to the movement of the pedestrian, the segmentation model can still stably extract the pedestrian body. In addition, the mask is adaptively adjusted as the pedestrian's posture and position change, avoiding the limitations of a fixed detection frame.

[0057] In one embodiment, step S4: preprocessing the segmented image area to extract image information corresponding to pedestrians, extracting and describing feature points for multiple pedestrians in the same image, and storing them as corresponding feature point information. This step is understood to address issues such as inaccurate trajectory prediction due to pedestrian motion and environmental changes, as well as unstable tracking and missed small target detection due to feature sparsity. It includes the following sub-steps:

[0058] S401, using the SuperPoint deep learning feature point detection model to perform feature detection on the preprocessed segmented image area to generate a feature point probability map; wherein the expression of the SuperPoint deep learning feature point detection model is:

[0059] P i,j,k =Softmax(Conv 1×1 (F) i,j,k ) (3)

[0060] in, P represents the feature point probability map, H and W represent the height and width of the original image, P i,j,k represents the probability value of the kth channel at position (i, j); the feature point probability map generated by this model contains 65 channels, 64 of which correspond to locations in an 8×8 local grid, and the remaining channel represents the category without feature points. It can be understood that by using the SuperPoint model to densely sample and generate a high-resolution feature point probability map, key feature points in the image are automatically detected, which can more accurately identify pedestrian features. Even in partially occluded pedestrians or low-resolution scenes, fine-grained feature points (such as clothing corner textures and backpack accessories) can still be detected, which means that a high recognition rate is always maintained, thereby improving the accuracy and robustness of monitoring.

[0061] S402: Based on the feature point probability map, extract the angle and color information surrounding the feature point to form descriptor information to describe the feature point. This descriptor information enhances the differentiation of feature points, enabling more accurate identification and association of the same pedestrian in subsequent inter-frame matching and tracking. When a pedestrian's appearance changes due to occlusion by a book or rapid movement, the descriptor maintains matching stability through local information, avoiding mismatches and trajectory drift.

[0062] S403: Extract feature points and descriptor information from the segmented image regions within each frame and store them as corresponding feature point information. This step ensures that pedestrian features in each frame are recorded in detail, enabling feature matching between consecutive frames and facilitating accurate tracking of each pedestrian across consecutive frames. This avoids the phenomenon of single-frame detection failing to utilize historical features, leading to tracking interruptions and improving the accuracy and real-time performance of pedestrian flow monitoring.

[0063] In one embodiment, step S5: establishing correspondences between pedestrians in motion based on feature point information and determining inter-frame motion results through the correspondences. It is understood that feature point information is used to accurately identify and track pedestrians in image sequences, thereby improving the accuracy and stability of pedestrian flow monitoring; and includes the following sub-steps:

[0064] S501. For key point pairs between consecutive frames, establish corresponding matching relationships using corner point information and feature descriptor information; by comparing the similarity of feature descriptors, select corresponding feature points between different frames. This can be understood as the process of more accurately identifying and tracking pedestrians between consecutive frames, maintaining high tracking accuracy even in dense crowds and occlusion situations, and effectively resolving tracking failures due to occlusion.

[0065] S502: Eliminate unmatched feature point information to remove invalid or inaccurate feature points, and add new feature point information to ensure that the number of feature points extracted from each frame reaches a preset threshold. As can be understood, eliminating incorrectly matched points prevents noise features from misleading trajectory prediction, while adding new points enhances trajectory continuity. This helps improve tracking stability and accuracy, particularly when dealing with small targets or pedestrians with unclear features, reducing missed detections and improving the accuracy of pedestrian flow monitoring.

[0066] S503. Based on the established matching relationships, the transformation relationship between each pair of feature points is calculated. By analyzing the transformation relationships between all matching feature points, the inter-frame motion results are comprehensively obtained. It can be understood that more accurate inter-frame motion parameters are obtained through joint optimization of multiple feature points (e.g., minimizing reprojection error). This can more accurately reflect the changes in pedestrian motion from one frame to the next, providing key data for pedestrian trajectory tracking and behavior analysis, and effectively solving the problem of pedestrian trajectory randomness.

[0067] In one embodiment, step S6: based on the inter-frame motion results, using Kalman filtering to predict the trajectory, and combining feature point extraction and corresponding matching relationships for verification, to obtain motion information of the same pedestrian in consecutive frames; understandably, using Kalman filtering to predict the pedestrian's motion trajectory and combining feature point extraction and matching relationships for verification to improve tracking accuracy and stability; includes the following sub-steps:

[0068] S601, defining a motion state and a measurement vector for a detection target in two consecutive frames of images;

[0069] The motion state is defined as:

[0070]

[0071] Among them, k represents the step length information per unit time, (x c ,y c ) represents the coordinates of the center point of the detection box, (w,h) represents the width and height of the detection box, v w 、v h Represents the velocity change in each dimension. Understanding this, by introducing multidimensional states such as size (w, h) and velocity in each dimension, combined with the segmented pedestrian region features (output of S3), the impact of background noise on motion estimation can be reduced. The introduction of velocity components enables the system to predict pedestrian acceleration, deceleration, or changes in direction, adapting to the complexity of pedestrian paths in library scenarios.

[0072] The measurement vector is defined as:

[0073] z k =Hx k +v k (5)

[0074] Where H represents the measurement matrix, v k represents the measurement noise, x k represents the state vector at time k.

[0075] Understandably, by precisely defining the motion state and measurement vector, the motion characteristics of pedestrians can be captured more accurately, thereby improving the accuracy of trajectory prediction.

[0076] S602: Predict and update the motion trajectory of the detection target at the next moment. The nonlinear equation is expressed as:

[0077] x k =f(x k-1 )+w k (6)

[0078] z k =h(xk )+v k (7)

[0079] Among them, f(·) and h(·) represent nonlinear functions, v k represents the measurement noise, w k represents process noise, z k Indicates the measurement result of the sensor.

[0080] Understandably, the motion trajectory of pedestrians is predicted and updated through nonlinear equations to adapt to the nonlinear characteristics of pedestrian motion, improve the flexibility and accuracy of prediction, and thus reduce tracking failures caused by randomness of motion direction.

[0081] S603: Based on the motion state of the detection target at the current moment and the true value point after Kalman filter approximation at the next moment, predict the state and covariance at the next moment, which is expressed as:

[0082]

[0083] Among them, in formula (8), represents the predicted state at time k, represents the predicted state at time k-1, and (8) represents the predicted state is the predicted state at time k-1 Obtained by applying the state transfer function f;

[0084] (9) represents the predicted covariance at time k, F k Represents the state transfer matrix, which is used to describe the linear influence of the state transfer function f on the state variables; represents the predicted covariance at time k-1; Indicates F k The transpose of , Q represents the process noise covariance matrix, which is used to describe the uncertainty of the model;

[0085] Formula (10) represents the state transfer matrix, F k It is the Jacobian matrix of the state transfer function f with respect to the state variable x, and predicts the state at the moment x equals k-1. It can be understood that the prediction and update mechanism of the Kalman filter can effectively integrate the current observation and prior knowledge, improve the accuracy and robustness of trajectory prediction, and thus solve the problem of trajectory prediction deviation caused by background dynamic changes.

[0086] In this way, through multi-dimensional state modeling (size, speed) and nonlinear optimization of Kalman filtering, the trajectory drift caused by dynamic background interference of the detection frame in traditional methods is reduced. In the scene of dense crowds and frequent occlusions in the library, the covariance matrix is ​​dynamically adjusted. and noise parameter Q to maintain tracking stability. Even if the pedestrian disappears briefly (such as squatting to pick up a book), the system can still predict its reappearance position based on its historical trajectory.

[0087] In one embodiment, step S7: based on the motion information and the association relationship, determining the unique detection information of the pedestrian, assigning a unique ID for storage, and using the unique ID to track each pedestrian entering and exiting to count the flow of people in the library. This ensures that each pedestrian can be accurately identified and tracked, and includes the following sub-steps:

[0088] S701: Integrate the correspondence between the inter-frame motion results and the feature point matching to determine the pedestrian identity association information. This improves the accuracy of pedestrian recognition and reduces tracking failures caused by occlusion or small targets.

[0089] S702: Detect uniqueness of pedestrians using the integrated motion information and association relationship. This ensures that each pedestrian can be accurately identified, avoiding misidentification due to similar features.

[0090] S703. For each pedestrian detected, a unique ID is assigned, and the ID is stored together with the corresponding motion information and association relationship; when a new pedestrian is detected, a new unique ID is assigned, and when the pedestrian leaves the monitoring area or disappears, the new pedestrian status is updated in real time and its ID is retained. Understandably, this ensures that the motion trajectory of each pedestrian is fully recorded to avoid trajectory breaks due to temporary occlusion or detection frame jitter (traditional methods rely on the detection module to reinitialize after the pedestrian disappears, resulting in discontinuous trajectories). It helps to accurately track each pedestrian in consecutive frames and maintain tracking continuity even when the pedestrian leaves the monitoring area or disappears. The retained and stored ID information can be used for subsequent analysis (such as hot spot area identification and abnormal behavior detection).

[0091] S704. Use a unique ID to track each pedestrian's entry and exit, and count the number of people in the library. As you can see, each pedestrian is counted only once (the ID is activated upon entry and deactivated upon exit), avoiding the statistical errors caused by ID confusion or duplicate detection in traditional methods. The unique ID mechanism ensures that each pedestrian's entry and exit is independently recorded, reducing missed detections and duplicate counting, and providing reliable data for traffic flow warnings and resource scheduling.

[0092] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] In one embodiment, a library passenger flow monitoring system based on multimodal and trajectory fusion is provided. The library passenger flow monitoring system based on multimodal and trajectory fusion corresponds one-to-one to the library passenger flow monitoring method based on multimodal and trajectory fusion in the above embodiment. Figure 2 As shown in the figure, the library traffic monitoring system based on multimodal and trajectory fusion includes:

[0094] The acquisition module 100 is used to acquire the video stream of the library and perform preprocessing to obtain the image to be detected containing weight information of the area to be detected;

[0095] The target detection module 200 is used to perform target detection on the image to be detected containing weight information and obtain the current pedestrian detection frame of the library;

[0096] The region segmentation module 300 is used to segment the original image based on the position of the pedestrian detection frame using a segmentation model to obtain segmented image regions;

[0097] The feature extraction module 400 is used to pre-process the segmented image area, extract image information corresponding to pedestrians, and extract and describe feature points for multiple pedestrians in the same image, and store them as corresponding feature point information;

[0098] The data association module 500 is used to establish a correspondence between pedestrians in motion based on feature point information and determine the inter-frame motion result through the correspondence;

[0099] The trajectory tracking module 600 is used to predict the trajectory based on the inter-frame motion results using Kalman filtering, and verify it by combining feature point extraction and corresponding matching relationships to obtain the motion information of the same pedestrian in consecutive frames;

[0100] The traffic statistics module 700 is used to determine the unique detection information of pedestrians based on movement information and association relationships, assign a unique ID for storage, and use the unique ID to track the entry and exit of each pedestrian to count the flow of people in the library.

[0101] Regarding the specific limitations of the library passenger flow monitoring system based on multimodal and trajectory fusion, please refer to the limitations of the library passenger flow monitoring method based on multimodal and trajectory fusion mentioned above, which will not be repeated here. The various modules in the above-mentioned library passenger flow monitoring system based on multimodal and trajectory fusion can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0102] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a library traffic flow monitoring method based on multimodality and trajectory fusion is implemented.

[0103] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0104] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A library traffic monitoring method based on multimodal and trajectory fusion, characterized by: include: S1, collect the video stream of the library and pre-process it to obtain the image to be detected containing weight information of the area to be detected; S2. Perform target detection on the image to be detected containing weight information to obtain the current pedestrian detection frame of the library; S3. Based on the position of the pedestrian detection frame, the original image is segmented using the segmentation model to obtain a segmented image area; S4. Preprocess the segmented image area to extract image information corresponding to the pedestrians, and extract and describe feature points for multiple pedestrians in the same image, and store them as corresponding feature point information respectively; S5. Based on the feature point information, establish a correspondence relationship between pedestrians in action, and determine the inter-frame motion result through the correspondence relationship; S6. Based on the inter-frame motion results, the trajectory is predicted using Kalman filtering, and verified by combining feature point extraction and corresponding matching relationships to obtain the motion information of the same pedestrian in consecutive frames; S7. Based on the motion information and association relationship, the unique detection information of the pedestrian is determined and assigned a unique ID for storage. The unique ID is used to track the entry and exit of each pedestrian to count the flow of people in the library.

2. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 1 is characterized in that: The step S1 further includes the following sub-steps: S101, using a front-end camera to collect a video stream of the area to be detected, extracting image frames from the video stream, and performing grayscale, equalization, and denoising processing on the image frames to obtain an image F to be detected; S102: performing weight assignment on the image to be detected F based on the attention mechanism to obtain weight information of the area to be detected; wherein the expression of the weight assignment method is: A=σ(Conv 1×1 ([F avg ;F max ])) (1) Where A∈R H×W represents the weight distribution of the associated image spatial information, H×W is the size information of the image, F avg and F max are channel dimension average pooling and maximum pooling respectively, and σ is the Sigmoid activation function.

3. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 2 is characterized in that: The step S2 further includes the following sub-steps: S201. Obtain a pedestrian dataset based on public datasets and private self-made datasets; S202, setting the ratio of the training set, test set, and validation set with a weight of 8:1:1 based on the pedestrian dataset, performing model training using the YOLOv11 algorithm, and obtaining the optimal training model weight; S203: Based on the optimal training model weights, the model is deployed to the terminal to perform pedestrian detection, and a detection result frame containing all pedestrian targets is output. Duplicate frames are sorted based on the non-maximum suppression model, and redundant frames are deleted to obtain pedestrian detection frames. The mathematical expression of the non-maximum suppression model is: Among them, β represents the set of all detected bounding boxes, b i and b j Represents two different bounding boxes, Area(b i ∩b j ) means calculating the intersection area of ​​two bounding boxes, Area(b i ∪b j ) represents the calculation of the union area of ​​two bounding boxes, θ NMS Represents the set threshold parameter; when the intersection over union (IoU) of two bounding boxes is less than the threshold θ NMS , it is confirmed that different targets are detected and will not suppress each other, and the two bounding boxes are retained.

4. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 3 is characterized in that: The step S3 further includes the following sub-steps: S301, obtaining position information of a pedestrian detection frame, and extracting a corresponding partial image from the original image according to the position of the pedestrian detection frame; S302: Input the local image into a pre-trained segmentation model for feature segmentation, extract inter-frame features of the target pedestrian, and generate mask information; S303, extracting a segmented image region of a pedestrian target from the original image using mask information, wherein the segmented image region includes a target region and a background region; And based on the mask information, count the number of all pedestrians in a single frame image.

5. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 4 is characterized in that: The step S4 further includes the following sub-steps: S401, using the SuperPoint deep learning feature point detection model to perform feature detection on the preprocessed segmented image area to generate a feature point probability map; wherein the expression of the SuperPoint deep learning feature point detection model is: P i,j,k =Softmax(Conv 1×1 (F) i,j,k ) (3) in, represents the feature point probability map, H and W represent the height and width of the original image, P i,j,k Represents the probability value of the kth channel at position (i, j); The feature point probability map generated by the model contains 65 channels, of which 64 channels correspond to the positions of the 8×8 local grid, and the other channel represents the category without feature points; S402, extracting angle information and color information around the feature point based on the feature point probability map to form descriptor information to describe the feature point; S403 : For the segmented image area in each frame of image, extract feature points and descriptor information respectively, and store them as corresponding feature point information respectively.

6. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 5 is characterized in that: The step S5 further includes the following sub-steps: S501: For key point pairs between consecutive frames, establish corresponding matching relationships using corner point information and feature descriptor information; and select corresponding feature points between different frames by comparing the similarity of feature descriptors. S502: Eliminate unmatched feature point information to remove invalid or inaccurate feature points, and add new feature point information to ensure that the number of feature points extracted from each frame of image reaches a preset threshold; S503: Based on the established matching relationship, the transformation relationship between each pair of feature points is calculated, and the inter-frame motion result is comprehensively obtained by analyzing the transformation relationships between all matching feature points.

7. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 6 is characterized in that: The step S6 further includes the following sub-steps: S601, defining a motion state and a measurement vector for a detection target in two consecutive frames of images; The motion state is defined as: Among them, k represents the step length information per unit time, (x c ,y c ) represents the coordinates of the center point of the detection box, (w,h) represents the width and height of the detection box, v w 、v h Respectively represent the velocity changes in each dimension; The measurement vector is defined as: z k =Hx k +v k (5) Where H represents the measurement matrix, v k represents the measurement noise, x k represents the state vector at time k; S602: Predict and update the motion trajectory of the detection target at the next moment. The nonlinear equation is expressed as: x k =f(x k-1 )+w k (6) z k =h(x k )+v k (7) Among them, f(·) and h(·) represent nonlinear functions, v k represents the measurement noise, w k represents process noise, z k Indicates the measurement results of the sensor; S603: Based on the motion state of the detection target at the current moment and the true value point after Kalman filter approximation at the next moment, predict the state and covariance at the next moment, which is expressed as: Among them, in formula (8), represents the predicted state at time k, represents the predicted state at time k-1, and (8) represents the predicted state is the predicted state at time k-1 Obtained by applying the state transfer function f; (9) represents the predicted covariance at time k, F k Represents the state transfer matrix, which is used to describe the linear influence of the state transfer function f on the state variables; represents the predicted covariance at time k-1; Indicates F k The transpose of , Q represents the process noise covariance matrix, which is used to describe the uncertainty of the model; (10) represents the state transfer matrix, F k It is the Jacobian matrix of the state transfer function f with respect to the state variable x, and predicts the state at the moment x equals k-1. Evaluated at.

8. The library traffic monitoring method based on multimodal and trajectory fusion according to claim 7 is characterized in that: The step S7 further includes the following sub-steps: S701, integrating the correspondence between the inter-frame motion results and the feature point matching to determine pedestrian identity association information; S702: Detect pedestrian uniqueness using the integrated motion information and association relationship. S703: For each detected pedestrian, a unique ID is assigned and stored together with the corresponding motion information and association relationship. When a new pedestrian is detected, a new unique ID is assigned. When the pedestrian leaves the monitoring area or disappears, the new pedestrian's status is updated and its ID is retained. S704. Use the unique ID to track the entry and exit of each pedestrian and count the flow of people in the library.

9. A library traffic monitoring system based on multimodal and trajectory fusion, characterized by: include: The acquisition module is used to acquire the video stream of the library and perform preprocessing to obtain the image to be detected containing weight information of the area to be detected; The target detection module is used to perform target detection on the image to be detected containing weight information and obtain the current pedestrian detection frame of the library; The region segmentation module is used to segment the original image based on the position of the pedestrian detection frame using the segmentation model to obtain the segmented image area; The feature extraction module is used to pre-process the segmented image area, extract the image information corresponding to the pedestrian, and extract and describe the feature points of multiple pedestrians in the same image, and store them as corresponding feature point information respectively; The data association module is used to establish the correspondence between pedestrians in action based on feature point information and determine the inter-frame motion results through the correspondence; The trajectory tracking module is used to predict the trajectory based on the inter-frame motion results using Kalman filtering, and verify it by combining feature point extraction and corresponding matching relationships to obtain the motion information of the same pedestrian in consecutive frames; The traffic statistics module is used to determine the unique detection information of pedestrians based on movement information and association relationships, and assign a unique ID for storage. The unique ID is used to track the entry and exit of each pedestrian to count the flow of people in the library.

Images