All-weather crowd statistical method based on thermal imaging camera and application of all-weather crowd statistical method
By adopting an all-weather population statistics method based on thermal imaging cameras in population density detection, combined with a cross-modal collaborative learning framework, the problem that the existing technology cannot identify people at night is solved, and high-precision population statistics and density information acquisition around the clock are achieved.
Patent Information
- Application Number
- CN202411964557.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-29
AI Technical Summary
The existing population density detection scheme cannot effectively identify people in night or in environments with insufficient light, resulting in a certain degree limiting the implementation of the population density detection algorithm.
The all-weather population statistics method based on the thermal imaging camera is adopted, and RGB three-channel images and T single-channel thermal infrared images are collected by integrating optical sensors and thermal imaging sensors, and input them into a cross-modal collaborative learning framework to perform feature extraction, head center point regression and candidate head point classification to achieve all-weather population statistics and density information acquisition.
It realizes effective crowd statistics during the day and at night, improves recognition accuracy and robustness, and can obtain personnel's location information and density information at the same time, which is suitable for multiple application fields.
Smart Images
Figure CN119904805A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image intelligent recognition and proposes an innovative all-weather crowd counting method based on a thermal imaging camera. Background Art
[0002] Crowd counting technology, also known as crowd density estimation, is a technology that uses computers to count or estimate the number of people in an image. By analyzing crowd gatherings, it can help city managers and organizers of large-scale events understand crowd congestion in real time, and is widely used in video surveillance and public safety.
[0003] In the 1960s and 1970s, the embryonic stage of crowd counting and recognition technology mainly relied on simple video surveillance systems. These systems were able to record and play back images, providing a basic means for security monitoring. However, the technology of this period relied entirely on manual monitoring and lacked automated analysis capabilities, resulting in low efficiency and failure to meet the needs of real-time processing of large amounts of crowd data.
[0004] With the development of digital technology in the 1980s and 1990s, crowd counting technology entered the digital stage. Video content was digitized, enabling digital storage and basic motion detection capabilities. This advancement significantly improved the efficiency of video storage and retrieval, and introduced automated motion detection that could trigger alarms. Despite this, motion detection technology was still relatively primitive, easily affected by environmental factors, and the accuracy of crowd counting was limited.
[0005] From the early 2000s to the early 2010s, primary intelligent video analysis technology began to emerge. This stage of technology achieved crowd counting and abnormal behavior detection through more advanced algorithms. Automated crowd counting and abnormal behavior identification significantly improved response speed and reduced reliance on manual monitoring. However, these algorithms have limited accuracy and robustness in complex environments and have high requirements for computing resources.
[0006] Since the mid-2010s, deep learning has greatly improved the accuracy of crowd counting and behavior recognition, significantly improving real-time analysis and response capabilities. However, the high demand for computing resources, data privacy and security issues of this technology have also become a focus of attention.
[0007] Methods based on deep learning technology have achieved good results, both in terms of accuracy and real-time performance. Currently, the mainstream methods are divided into four categories:
[0008] Thanks to the rapid development of target detection and face detection technology, early methods used target detection or face detection algorithms to identify people in pictures, thereby obtaining the location information and number of people. This method has high recognition accuracy and rarely has recognition errors, but the recall rate is extremely low. The places where the number of people needs to be counted are often areas where the crowds are highly concentrated and overlapping. The field of view of the picture is large, and the proportion of each person in the image pixels is very small, which is not conducive to target detection or face detection. Small targets and high overlap make the results obtained using this method often unsatisfactory.
[0009] Based on the method of number regression, this method directly predicts the number of people in the image. Compared with the detection-based method, the accuracy has been greatly improved, but it cannot obtain the location information and dense areas of people in the image, and cannot discover and locate areas with hidden dangers early.
[0010] Based on the density map estimation method, since the data set used for crowd statistics is a point cloud map formed by the point coordinates of each person's head, the point cloud map is too sparse compared to the entire image, which is not conducive to network learning. Based on the density map estimation method, the density map obtained after Gaussian convolution of the point cloud map is relatively dense, which is convenient for network learning and greatly improves the recognition accuracy. At the same time, dense areas can also be obtained, but the obtained density map cannot directly obtain the location information of people on the original image.
[0011] Based on point coordinate regression, the coordinates of each person’s head point are directly regressed, and the point coordinates are classified to determine whether they belong to the background. This not only solves the positioning problem, but also allows the density to be intuitively felt on the original image. However, it cannot process night pictures, and the model can only input RGB three-channel images.
[0012] In summary, the existing crowd density detection scheme has some defects, especially the inability to handle nighttime recognition scenes, which to some extent limits the implementation of crowd density detection algorithms. Summary of the invention
[0013] In response to the shortcomings of the existing technology, we disclose an all-weather crowd counting method based on thermal imaging cameras and its application, which can not only identify people around the clock, but also obtain the location and density information of people.
[0014] The present invention includes the following technical solutions:
[0015] A method for counting people around the clock based on a thermal imaging camera comprises the following steps:
[0016] S1 uses a multispectral camera that integrates optical sensors and thermal imaging sensors to collect RGB three-channel images and T single-channel thermal infrared images of the scene;
[0017] S2 inputs the collected RGBT four-channel data into the cross-modal collaborative learning framework, which consists of multiple modality-specific branches to process and fuse the RGBT data;
[0018] S3 extracts features from the fused data through the feature extraction module in the framework;
[0019] Based on the extracted features, S4 predicts the position of the head center point in the crowd through the head center point regression branch, and classifies the possible head center points through the candidate head point classification branch to distinguish the real head from the background or other interference factors;
[0020] S5 counts the total number of people based on the predicted head center position and classification results.
[0021] Furthermore, the above-mentioned all-weather crowd counting method based on thermal imaging camera,
[0022] Step S2 also includes preprocessing the RGB image and T image data, including standardization, wherein each pixel value in the RGB image is calculated by the standardized pixel value, and the single-channel thermal infrared data in the T image is mapped to the three RGB channels, each channel representing a different temperature range, and the algorithm is as follows:
[0023] ①RGB
[0024] For each pixel value (R, G, B) in an RGB image, the normalized pixel value is calculated as follows:
[0025]
[0026] ②T
[0027] The single-channel thermal infrared data is mapped to three channels of RGB. Each channel represents a different temperature range and indicates the temperature difference by the change of color. Therefore, for each pixel value (R, G, B) in the T image, the standardized pixel value is calculated as follows:
[0028]
[0029] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging cameras, in step S2, the cross-modal collaborative learning framework includes three parallel backbone networks and an information aggregation and distribution module, wherein the top and bottom backbone networks respectively extract features of RGB images and T single-channel thermal infrared images, the middle backbone network extracts common features across modalities, and the information aggregation and distribution module is responsible for coordinating and integrating information from the three backbone networks to achieve feature fusion.
[0030] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging cameras, the information aggregation and distribution module enhances the collaboration of modal-specific and modal-shared representations by dynamically transmitting specific shared information, specifically including an information aggregation transmission mechanism and an information distribution transmission mechanism to achieve the complementarity of multimodal data.
[0031] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging camera, the function of the information aggregation and distribution module is realized by the following formula:
[0032]
[0033] IADM is a multimodal information aggregation and distribution module, r and t represent the RGB branch and T branch respectively, s is the fusion branch, and i and j represent the number of the convolutional layer. After the features of different branches pass through the IADM module, they further learn advanced multimodal representations.
[0034] The information aggregation and transfer mechanism aggregates the contextual information of all modality-specific features to enhance the modality-shared features, and applies two gating functions to adaptively propagate complementary information to re-refine the modality-shared features. Enhanced Features The formula is as follows:
[0035]
[0036] The gating function is implemented by the convolutional layer. and For the gated weights, through such a multiplication mechanism, complementary information is effectively embedded into the modality shared representation;
[0037] The information distribution transmission mechanism distributes the information of the shared features of the new modality after information aggregation, and refines the unique features of each feature. Similar to the information aggregation transmission mechanism, there are two gating functions for information filtering. The specific calculation formula is as follows:
[0038]
[0039] All enhanced features Passed to the next layer in each branch for further learning.
[0040] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging camera, in step S4, the shape of the head center point regression branch output is (b, h*w, 2), where b is the batch-size, h and w are the height and width of the feature map respectively, and this branch is responsible for predicting the x and y coordinate offsets of the target in each grid unit.
[0041] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging camera, in step S4, the shape of the output of the candidate head point classification branch is (b, h*w, 1), and the branch uses the sigmoid activation function to convert the output into a probability value, indicating the probability that the target in each grid unit is a human head.
[0042] Furthermore, in the above-mentioned all-weather crowd counting method based on thermal imaging camera, in step S5, the statistics of the total number of people are based on the Gaussian distribution formula, taking into account the distance between each detected center point and the nearest head center point, and a parameter λ for adjusting the Gaussian distribution width, the formula is as follows:
[0043]
[0044] Among them, C is the total number of people, D is i is the distance from the center point of the i-th head to the center point of the nearest head, λ is a parameter used to adjust the width of the Gaussian distribution, and n is the total number of detected head pixels.
[0045] The present invention also discloses a hardware device of an all-weather crowd counting system based on a thermal imaging camera, comprising:
[0046] 1) A multispectral camera integrated with an optical sensor and a thermal imaging sensor to simultaneously capture RGB three-channel images and T single-channel thermal infrared images of the scene;
[0047] 2) at least one processor, configured to perform data processing, feature extraction, head center point regression prediction, candidate head point classification, and crowd total count in a cross-modal collaborative learning framework;
[0048] 3) a storage unit, used to store the model parameters, processed image data and statistical results of the cross-modal collaborative learning framework;
[0049] 4) Communication interface, used for data exchange and communication with external devices or systems;
[0050] 5) Optionally, one or more domestic chips are also included, including but not limited to Baidu Kunlun chips, which are used to optimize and accelerate the operation of the cross-modal collaborative learning framework and improve the efficiency and performance of real-time analysis of crowd density.
[0051] The present invention also discloses an application of an all-weather crowd counting method based on a thermal imaging camera, which is applied to any of the following fields 1)-5):
[0052] 1) Public safety monitoring: Real-time monitoring of crowd density at large events, transportation hubs and other places to prevent crowding and stampedes and ensure public safety;
[0053] 2) Retail analysis: In retail places such as shopping malls and supermarkets, customer flow and distribution are analyzed through crowd statistics to optimize product display and promotion strategies;
[0054] 3) Traffic planning: In urban roads, public transportation and other transportation systems, monitor the flow of people and vehicles, guide traffic signal control and route planning, and improve traffic efficiency;
[0055] 4) Emergency response: In emergency situations such as natural disasters and emergencies, quickly assess the size of the crowd, guide the allocation of rescue resources and plan evacuation routes;
[0056] 5) Smart City: Combined with other smart city technologies, such as big data analysis and artificial intelligence, it provides data support and decision-making basis for urban planning, public resource allocation and emergency management.
[0057] In summary, the present invention proposes an all-weather crowd counting method based on thermal imaging cameras and its application. The method combines the data of optical sensors and thermal imaging sensors, and realizes the fusion and processing of RGBT four-channel data through a cross-modal collaborative learning framework, so that crowd counting can be performed all-weather and the location information and density information of people can be obtained at the same time. The following is a detailed description of the beneficial effects of the present invention:
[0058] 1All-weather recognition capability:
[0059] The present invention integrates optical sensors and thermal imaging sensors to effectively collect image data both during the day and at night. Thermal imaging sensors are not limited by lighting conditions and can capture the heat emitted by the human body at night or in low-light environments, thereby ensuring all-weather crowd counting capabilities.
[0060] 2. Improve recognition accuracy and robustness:
[0061] The cross-modal collaborative learning framework can fuse the information of RGB images and thermal infrared images, and use the complementarity of the two to improve recognition accuracy. RGB images provide rich color and texture information, while thermal infrared images provide human body heat distribution information. The combination of the two can more accurately identify the position and number of human heads.
[0062] Through the steps of feature extraction, head center point regression and candidate head point classification, the present invention can accurately distinguish the real head from the background or other interference factors, thereby improving the accuracy and robustness of statistics.
[0063] 3. Get personnel location information and density information:
[0064] The present invention not only counts the total number of people, but also predicts the position of the center point of the head in the crowd through the center point regression branch of the head, thereby realizing the accurate positioning of the position of the person.
[0065] The crowd total counting method based on the Gaussian distribution formula can take into account the distance between each detected center point and the center point of the nearest head, and more accurately reflect the crowd density distribution.
[0066] 4Wide range of applications:
[0067] The present invention can be applied to many fields such as public safety monitoring, retail analysis, traffic planning, emergency response and smart cities, providing data support and decision-making basis for urban planning, public resource allocation and emergency management.
[0068] 5Technological advancement and innovation:
[0069] The present invention adopts advanced technologies such as cross-modal collaborative learning framework, dynamic transmission of specific shared information, head center point regression and candidate head point classification to achieve innovation in crowd counting methods.
[0070] By integrating domestic chips (such as Baidu Kunlun Core) to optimize and accelerate the operation of the framework, the efficiency and performance of real-time analysis of crowd density have been improved, demonstrating the cutting-edge nature and practicality of the technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 Overall algorithm network structure diagram;
[0072] Figure 2 Information aggregation and distribution algorithm structure diagram;
[0073] Figure 3 Neck pyramid network structure diagram;
[0074] Figure 4 Overall framework diagram;
[0075] Figure 5 Schematic diagram of RGBT framework for rapid assessment of crowd size;
[0076] Figure 6 Schematic diagram of data collection in bright environment;
[0077] Figure 7 Figure 1 shows the data collection in a dim environment;
[0078] Figure 8 Example 2 Model conversion output results;
[0079] Fig. 9 Implementation effect diagram of Example 2. DETAILED DESCRIPTION
[0080] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] Example:
[0082] The design of the all-weather crowd counting method based on thermal imaging cameras is as follows: In terms of hardware, two sensors, an optical sensor and a thermal imaging sensor, are integrated in one camera. The optical sensor outputs RGB three-channel images, and the thermal imaging camera outputs T single-channel images. In terms of the model, a cross-modal collaborative learning framework is introduced, which consists of multiple specific modal branches. It can process the thermal imaging data of the RGBT four-channel fusion. After the backbone feature extraction, point regression and point classification are performed, which can provide high-quality detection results regardless of day or night.
[0083] The overall process is:
[0084] (1) A multispectral camera is used to provide color images of the scene, which is suitable for daily visual observation and image processing. The thermal infrared sensor captures the thermal radiation emitted by the object and generates a temperature map, which can be used for night vision.
[0085] (2) Through the input RGBT four-channel data, the channel information is extracted and fused so that it can fully integrate the multi-modal complementarity. The fused feature map is identified to locate the location of people and crowd density information, and the density can be directly reflected on the original image.
[0086] The algorithm network structure diagram is as follows Figure 1 As shown:
[0087] The details are as follows:
[0088] 1.RGBT data access
[0089] RGBT data stream refers to a data stream that combines visible light (RGB) and thermal infrared (T) images. This data stream is particularly important in multiple recognition fields because it can provide multimodal information and enhance the robustness of target tracking. RGB images provide rich color and texture information, but their recognition capabilities are limited in poor lighting conditions or backgrounds with similar textures. In contrast, thermal infrared images are not affected by lighting conditions and can capture the thermal radiation of objects, allowing them to clearly identify targets even in foggy days or at night. This complementarity gives the RGBT data stream stronger recognition capabilities in complex environments.
[0090] In the RGBT data stream, due to device limitations and alignment errors, the images of the two modalities are not strictly aligned. Therefore, the annotation in the data stream needs to take this misalignment into account, and thermal imaging is usually used as the main modality for annotation because its effect is more stable. We use max-pooling in the network layer. Max-pooling has translation invariance. By extracting the maximum value of the local area, it can offset the misalignment of the RGB-thermal image to a certain extent.
[0091] The distribution of RGB and T image data is different. We use different means and standard deviations to process RGB and T data respectively:
[0092] ①RGB
[0093] For each pixel value (R, G, B) in an RGB image, the normalized pixel value is calculated as follows:
[0094]
[0095] ②T
[0096] The single-channel thermal infrared data is mapped to three channels of RGB, each of which represents a different temperature range and represents the temperature difference through color changes. Therefore, for each pixel value (R, G, B) in the T image, the standardized pixel value is calculated as follows:
[0097]
[0098] 2. Multimodal Data Fusion
[0099] The proposed RGBT crowd counting framework is an innovative multimodal learning system that combines the rich color information of RGB images and the thermal infrared data of thermal images to improve the accuracy and robustness of crowd counting. The framework consists of three parallel backbone networks and an information aggregation and distribution module. The top and bottom backbones are developed for modality-specific (RGB images and thermal images) representation learning, respectively extracting features of their respective modalities. The middle backbone focuses on modality-shared representation learning and extracts common features across modalities. This design makes full use of the complementarity of multimodal data and achieves collaborative enhancement of modality-specific and modality-shared representations by dynamically transmitting specific shared information.
[0100] The information aggregation and distribution module is the core of the framework, responsible for coordinating and integrating information from the three backbone networks to achieve deep feature fusion. The design of this module allows the system to dynamically transfer information between specific modalities and shared modalities to enhance the expressiveness of features. Ultimately, the modality-shared features contain comprehensive information and help generate high-quality feature maps; the algorithm structure is as follows Figure 2 shown.
[0101]
[0102] Among them, IADM is a multimodal information aggregation and distribution module, r and t represent the RGB branch and T branch respectively, s is the fusion branch, and i and j represent the number of the convolutional layer. After the features of different branches pass through the IADM module, advanced multimodal representation is further learned.
[0103] The left figure shows the information aggregation and transmission mechanism, which aggregates the contextual information of all modality-specific features to enhance the modality-shared features, and applies two gating functions to adaptively propagate complementary information to re-refine the modality-shared features. Enhanced Features The formula is as follows:
[0104]
[0105] The gating function is implemented by the convolutional layer. and is the gated weight. Through such a multiplication mechanism, the complementary information is effectively embedded into the modality shared representation.
[0106] The right figure shows the information distribution transmission mechanism. After information aggregation, the information of the shared features of the new modality is distributed, and the unique features of each feature are refined respectively. Similar to the information aggregation transmission mechanism, there are two gating functions for information filtering. The specific calculation formula is as follows:
[0107]
[0108] Finally, all enhanced features Passed to the next layer in each branch for further learning.
[0109] Due to the data fusion of the IADM module, the complementary information of the input RGB image and the thermal image is gradually transferred to the modality-shared representation. After obtaining the feature map that fuses the RGB and T data, the framework uses two branches to further process the data. The head center point regression branch is responsible for directly predicting the position of the head center point in the crowd, providing accurate positioning information for crowd counting. The candidate head point classification branch classifies possible head center points, distinguishes the real head from the background or other interference factors, and thus improves the accuracy of counting.
[0110] 3. Network Architecture
[0111] 1) Backbone
[0112] It adopts an architecture similar to the front-end block of VGG16, which contains a total of 5 blocks. The IADM multimodal information aggregation and distribution module is inserted into each block level. IADM dynamically transmits complementary information between modality-specific features and modality-shared features to achieve mutual enhancement. The three branches of RGT, T, and S (fusion) are all composed of the above structure. The specific network structure is shown in Table 1 below. The 5 blocks are C1, C2, C3, C4, and C5 respectively:
[0113] Table 1 Network structure
[0114]
[0115]
[0116] 2) neck
[0117] The feature pyramid network is used to combine feature information of different scales to improve the detection ability of the model. The specific network structure is Figure 3 shown.
[0118] 3)head
[0119] The head part contains two branches, the classification branch and the point regression branch. The regression branch is responsible for predicting the x and y coordinates of the point, and the classification branch is responsible for judging whether the predicted point belongs to the head. Assuming that the final prediction image size is h and w, and the batch-size is b, the shape of the point regression branch output is (b, h*w, 2), and the shape of the classification branch output is (b, h*w, 1). The network structure is shown in Table 2:
[0120] Table 2 Head network structure
[0121]
[0122] The point regression calculation formula can be expressed as:
[0123] Regression_Output=Regression(Feature_Map)
[0124] Regression represents the regression branch, Feature_Map is the feature map output by neck, and Regression_Output is the output of the regression branch, which contains the x and y coordinate offsets of the target in each grid cell.
[0125] The classification branch calculation formula can be expressed as:
[0126] Classification_Output=Sigmoid(Classification(Feature_Map))
[0127] Classification represents the classification branch, Feature_Map is the feature map in the network, Sigmoid is the sigmoid activation function, which converts the output of the classification branch into a probability value, and Classification_Output is the output of the classification branch, which represents the probability that the target in each grid cell is a human head.
[0128] 4. Crowd Counting Algorithm
[0129] We designed the following formula to count the total number of people:
[0130]
[0131] Among them, C is the total number of people, D is i is the distance from the center point of the i-th head to the center point of the nearest head, λ is a parameter used to adjust the width of the Gaussian distribution, and n is the total number of detected head pixels. This formula is based on the Gaussian distribution and is used to estimate whether each detected center point corresponds to an independent individual, thereby calculating the total number of people. This method can effectively reduce the error caused by crowd overlap and improve the accuracy of counting.
[0132] 5. Implementation Details
[0133] 1) Dataset
[0134] We use the RGBT-CC dataset for training. This dataset uses RGBT images taken by optical imagers in places such as shopping malls, streets, playgrounds, train stations, and subway stations. It contains a total of 2030 pairs of representative RGBT images. Among these samples, 1013 pairs are collected under light and 1017 pairs are collected in the dark, with a total of 138,389 head point coordinates, with an average of 68 people per image.
[0135] 2) Training details
[0136] The Adam algorithm with an intrinsic learning rate of 1e-4 was used to optimize the model parameters. No pre-trained model was used. The network parameters were randomly initialized for training. The batch-size was set to 1, and a total of 3500 epochs were trained.
[0137] 6. Overall framework
[0138] The overall framework is as follows Figure 4 shown.
[0139] The technical advantages of this RGBT crowd counting framework lie in its robustness, accuracy, real-time and scalability. It can work stably under different lighting and environmental conditions, providing high-quality feature maps and accurate head center point prediction. Parallel processing and optimized algorithm design enable the framework to achieve real-time crowd counting. In addition, the design of the framework allows for further integration of other modal data, such as depth information, to further improve performance.
[0140] The application scenarios are wide-ranging, including public safety monitoring, retail analysis, traffic planning, and emergency response. In public safety monitoring, the RGBT framework can monitor crowd density at large events or transportation hubs to prevent crowding and stampedes. Traffic planning can monitor the use of public transportation and guide traffic planning and scheduling. In emergencies, the RGBT framework can quickly assess the size of the crowd and guide the allocation of rescue resources.
[0141] Furthermore, the performance of our proposed algorithm and CSRNet algorithm under different lighting conditions of RGBT-CC, single-modal data is directly input into CSRNet, and multi-modal uses our framework, the comparison is shown in Table 3.
[0142] Table 3 Parameters under different lighting conditions.
[0143]
[0144] Data collection in bright environment is shown below: Figure 6 As shown, data collection in dim environment shows Figure 7 .
[0145] The root mean square error (RMSE) is used as the evaluation metric, and the grid mean absolute error (GAME) is used to evaluate the performance of different regions. For a specific level l, the image is divided into four l Non-overlapping areas, and calculate the counting error of each area, the GAME of the lth layer is calculated as:
[0146]
[0147] Where N is the total number of test samples For the i th Image j th The estimated counts and corresponding GT counts for each region.
[0148] Example 2
[0149] This algorithm (Example 1) has been successfully transplanted to domestic chips and deployed to large stations for real-time analysis of crowd density. The domestic chip selected is Baidu Kunlun Core. The model is transplanted using the tools provided by Kunlun Core. The training data uses RGBT-CC. The overall development work is shown in Table 5:
[0150] Table 4 Overall development process
[0151] Serial number Work 1 Kunlun core algorithm SDK and environment deployment 2 Training using the RGBT-CC dataset 3 Model conversion and migration 4 Interface packaging test 5 On-site deployment
[0152] After the model conversion is successful, the output content is as follows Figure 8 shown.
[0153] Use the torch model before conversion and the klx model after conversion for accuracy comparison:
[0154] Table 5 Accuracy comparison
[0155] Evaluation indicators torch klx MAE 9.44 9.48
[0156] 200 images were tested on the Kunlun chip, using different batch-sizes. The accuracy, average inference time, and video memory usage are as follows:
[0157] Table 6 Video memory usage
[0158]
[0159]
[0160] The actual effect is as follows Fig. 9 shown.
[0161] From the above potential application scenarios, we can see that
[0162] The algorithm we provide can not only be quickly ported to various domestic and foreign chips, but also monitor crowd density around the clock, especially in low-light environments.
[0163] For large-scale events such as concerts, sporting events and festivals, 24 / 7 crowd counting technology is key to ensuring event safety and improving the event experience. The technology can monitor crowd density in real time to prevent crowding and stampedes and ensure the safety of participants. In emergency situations, thermal imaging technology can quickly locate the crowd and guide effective evacuation. In addition, by analyzing the behavior and flow patterns of event participants, organizers can optimize event layout and process to enhance the experience of participants.
[0164] In the construction of smart cities, all-weather crowd counting technology based on thermal imaging cameras plays an important role. This technology can optimize traffic flow management and reduce urban congestion by monitoring the flow of pedestrians and vehicles in real time. In addition, thermal imaging data can help city planners to rationally plan public facilities based on crowd distribution and flow patterns, and improve the efficiency of urban resource allocation.
[0165] In crowded areas such as large stations and tourist attractions, this method can monitor and count the density of people in real time, providing accurate data support for management. By analyzing and visualizing crowd flow data, it is possible to effectively control crowd flow order, reduce congestion and traffic accidents, and improve the safety and convenience of people's travel.
[0166] challenge:
[0167] Thermal imaging cameras do have significant advantages in all-weather crowd density analysis, especially in large stations and crowded scenic spots. However, the cost of thermal imaging cameras is a challenge that cannot be ignored. The following are several main reasons why thermal imaging cameras are expensive:
[0168] 1) There are few materials that can transmit infrared light. Only a few rare metals in nature can achieve a transmittance of more than 95%. Artificial synthesis technology is not yet mature, which increases material costs.
[0169] 2) Although thermal imaging technology has potential applications in many fields, the current market size is relatively small and large-scale demand has not yet formed, resulting in low production capacity and high cost per product.
[0170] Despite these challenges, market analysis shows that with the advancement of technology and the reduction of costs, the application market of thermal imaging cameras is expected to expand further, and the market prospects are promising. With the breakthroughs of domestic manufacturers in detector technology and the increase in production capacity, it is expected that the cost of thermal imaging cameras will be reduced in the future, and the application areas will be further expanded.
[0171] The above are limited to several preferred embodiments of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A method for counting people around the clock based on a thermal imaging camera, characterized in that: The following steps are involved: S1 uses a multispectral camera that integrates optical sensors and thermal imaging sensors to collect RGB three-channel images and T single-channel thermal infrared images of the scene; S2 inputs the collected RGBT four-channel data into the cross-modal collaborative learning framework, which consists of multiple modality-specific branches to process and fuse the RGBT data; S3 extracts features from the fused data through the feature extraction module in the framework; Based on the extracted features, S4 predicts the position of the head center point in the crowd through the head center point regression branch, and classifies the possible head center points through the candidate head point classification branch to distinguish the real head from the background or other interference factors; S5 counts the total number of people based on the predicted head center position and classification results.
2. The all-weather crowd counting method based on a thermal imaging camera according to claim 1, characterized in that: Step S2 also includes preprocessing the RGB image and T image data, including standardization, wherein each pixel value in the RGB image is calculated by the standardized pixel value, and the single-channel thermal infrared data in the T image is mapped to the three RGB channels, each channel representing a different temperature range, and the algorithm is as follows: ①RGB For each pixel value (R, G, B) in an RGB image, the normalized pixel value is calculated as follows: The single-channel thermal infrared data is mapped to three channels of RGB. Each channel represents a different temperature range and indicates the temperature difference by the change of color. Therefore, for each pixel value (R, G, B) in the T image, the standardized pixel value is calculated as follows:
3. The all-weather crowd counting method based on a thermal imaging camera according to claim 1, characterized in that: In step S2, the cross-modal collaborative learning framework includes three parallel backbone networks and an information aggregation and distribution module, wherein the top and bottom backbone networks extract the features of RGB images and T single-channel thermal infrared images respectively, the middle backbone network extracts common features across modalities, and the information aggregation and distribution module is responsible for coordinating and integrating information from the three backbone networks to achieve feature fusion.
4. The all-weather crowd counting method based on a thermal imaging camera according to claim 3, characterized in that: The information aggregation and distribution module enhances the collaboration of modality-specific and modality-shared representations by dynamically transmitting specific shared information, specifically including an information aggregation delivery mechanism and an information distribution delivery mechanism, so as to achieve complementarity of multimodal data.
5. The all-weather crowd counting method based on thermal imaging camera according to claim 4, characterized in that: The function of the information aggregation and distribution module is implemented by the following formula: IADM is a multimodal information aggregation and distribution module, r and t represent the RGB branch and T branch respectively, s is the fusion branch, and i and j represent the number of the convolutional layer. After the features of different branches pass through the IADM module, they further learn advanced multimodal representations. The information aggregation and transfer mechanism aggregates the contextual information of all modality-specific features to enhance the modality-shared features, and applies two gating functions to adaptively propagate complementary information to re-refine the modality-shared features. Enhanced Features The formula is as follows: The gating function is implemented by the convolutional layer. and For the gated weights, through such a multiplication mechanism, complementary information is effectively embedded into the modality shared representation; The information distribution transmission mechanism distributes the information of the shared features of the new modality after information aggregation, and refines the unique features of each feature. Similar to the information aggregation transmission mechanism, there are two gating functions for information filtering. The specific calculation formula is as follows: All enhanced features Passed to the next layer in each branch for further learning.
6. The all-weather crowd counting method based on thermal imaging camera according to claim 1, characterized in that: In step S4, the shape of the head center point regression branch output is (b, h*w, 2), where b is the batch-size, h and w are the height and width of the feature map, respectively. This branch is responsible for predicting the x and y coordinate offsets of the target in each grid unit.
7. The all-weather crowd counting method based on thermal imaging camera according to claim 1, characterized in that: In step S4, the shape of the output of the candidate head point classification branch is (b, h*w, 1). This branch uses the sigmoid activation function to convert the output into a probability value, indicating the probability that the target in each grid cell is a human head.
8. The all-weather crowd counting method based on a thermal imaging camera according to claim 1, characterized in that: In step S5, the statistics of the total number of people are based on the Gaussian distribution formula, taking into account the distance between each detected center point and the nearest head center point, and a parameter λ for adjusting the width of the Gaussian distribution. The formula is as follows: Among them, C is the total number of people, D is i is the distance from the center point of the i-th head to the center point of the nearest head, λ is a parameter used to adjust the width of the Gaussian distribution, and n is the total number of detected head pixels.
9. A hardware device for an all-weather crowd counting system based on a thermal imaging camera, characterized in that: include: 1) A multispectral camera integrated with an optical sensor and a thermal imaging sensor to simultaneously capture RGB three-channel images and T single-channel thermal infrared images of the scene; 2) at least one processor, configured to perform data processing, feature extraction, head center point regression prediction, candidate head point classification, and crowd total count in a cross-modal collaborative learning framework; 3) a storage unit, used to store the model parameters, processed image data and statistical results of the cross-modal collaborative learning framework; 4) Communication interface, used for data exchange and communication with external devices or systems; 5) Optionally, one or more domestic chips are also included, including but not limited to Baidu Kunlun chips, which are used to optimize and accelerate the operation of the cross-modal collaborative learning framework and improve the efficiency and performance of real-time analysis of crowd density.
10. An application of an all-weather crowd counting method based on a thermal imaging camera, characterized in that: This method is applied to any of the following areas: 1) Public safety monitoring: Real-time monitoring of crowd density at large events, transportation hubs and other places to prevent crowding and stampedes and ensure public safety; 2) Retail analysis: In retail places such as shopping malls and supermarkets, customer flow and distribution are analyzed through crowd statistics to optimize product display and promotion strategies; 3) Traffic planning: In urban roads, public transportation and other transportation systems, monitor the flow of people and vehicles, guide traffic signal control and route planning, and improve traffic efficiency; 4) Emergency response: In emergency situations such as natural disasters and emergencies, quickly assess the size of the crowd, guide the allocation of rescue resources and plan evacuation routes; 5) Smart City: Combined with other smart city technologies, such as big data analysis and artificial intelligence, it provides data support and decision-making basis for urban planning, public resource allocation and emergency management.