Indoor people counting method and system based on multi-element data fusion

Abstract

The application discloses a kind of based on multi-element data fusion's indoor personnel counting method and system, belong to indoor personnel image target detection.The application is by to RGB indoor personnel occupancy graph, RGB indoor environment background graph, infrared indoor personnel occupancy graph and infrared indoor environment background graph carry out depth slow feature analysis, obtain RGB silhouette graph and infrared silhouette graph;RGB silhouette graph and infrared silhouette graph are respectively input space-time attention mechanism module and are handled, obtain RGB space-time feature and infrared space-time feature;According to RGB space-time feature and infrared space-time feature, obtain aggregated feature;Aggregated feature is convoluted and handled, and the image containing head target detection frame is output, the quantity of head target detection frame is calculated, and the quantity of indoor personnel is obtained.Can enhance personnel feature information, improve indoor personnel detection precision;Solved the prior art in complex indoor scene, head is easily shielded and misdetected and the problem that personnel moves leads to inaccurate head detection.

Description

Technical Field

[0001] This invention relates to the field of indoor people image target detection technology, and in particular to an indoor people counting method and system based on multi-source data fusion. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Occupancy rate information plays a crucial role in building energy conservation and maintaining a comfortable indoor environment. Utilizing occupancy rate data to adjust the operation of HVAC or lighting systems can ensure resident comfort while achieving energy savings and emission reductions. Studies have shown that control strategies based on occupancy rate information can save approximately 20% to 45% of building energy. To accurately detect building occupancy rates, researchers have proposed various solutions, including monitoring using smart meters, environmental sensors, cameras, and Wi-Fi. Compared to other methods, vision-based occupancy rate acquisition methods offer better real-time performance and detection accuracy. This method typically uses cameras to capture images and videos, then applies image processing, video analysis, and deep learning techniques to detect building occupancy rates.

[0004] Personnel detection methods are mainly divided into three categories: body detection, face detection, and head detection. Body detection and face detection methods have relatively obvious limitations, while head detection, due to the fact that the human head is not easily obscured and its outline is relatively clear, has always been the focus of personnel detection in complex indoor scenes.

[0005] While many head-detection-based indoor person detection methods have made significant progress, existing methods still have the following limitations in more complex indoor scenarios:

[0006] (1) Many background objects, such as backpacks, flower pots and portraits, which are similar in size, color and texture to the shape of a human head, have a certain probability of being misdetected as human heads, resulting in the number of detected heads not matching the actual number of heads.

[0007] (2) Since indoor scenes are mostly small in scale and crowded, the obtained image data will have certain occlusion problems, and it is difficult to detect heads with high confidence in this scene.

[0008] (3) The movement of people can also cause changes in the size, posture, texture and lighting of the head. There is a certain probability that these moving heads will not be detected, resulting in the number of detected heads being lower than the actual number of heads. Summary of the Invention

[0009] In order to solve the problems in the prior art, the present application provides an indoor personnel counting method and system based on multi-element data fusion, electronic equipment and computer readable storage medium, which combines multi-element data fusion, deep slow feature analysis, space-time attention mechanism and head recognition to count indoor personnel, improves target detection progress, and further improves the accuracy of indoor personnel counting.

[0010] In a first aspect, the present application provides an indoor personnel counting method based on multi-element data fusion.

[0011] The indoor personnel counting method based on multi-element data fusion comprises the following steps.

[0012] An RGB image and an infrared image are obtained, wherein the RGB image comprises an RGB indoor personnel occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor personnel occupancy map and an infrared indoor environment background map.

[0013] The RGB image and the infrared image are input into a preset indoor personnel counting model for processing to obtain the number of indoor personnel.

[0014] Deep slow feature analysis is performed on the RGB image and the infrared image to obtain an RGB silhouette map and an infrared silhouette map.

[0015] The RGB silhouette map and the infrared silhouette map are input into a space-time attention mechanism module for processing to obtain RGB space-time features and infrared space-time features.

[0016] The aggregated features are obtained according to the RGB space-time features and the infrared space-time features.

[0017] The aggregated features are subjected to convolution processing to output an image containing a head target detection frame, the number of head target detection frames is calculated, and the number of indoor personnel is obtained.

[0018] Further, the deep slow feature analysis on the RGB image comprises the following steps.

[0019] The RGB indoor personnel occupancy map and the RGB indoor environment background map in the RGB image dataset are input into a variational autoencoder respectively to obtain RGB personnel occupancy features and RGB environment features.

[0020] The RGB personnel occupancy features and the RGB environment features are input into a slow feature analysis model for processing to obtain the RGB silhouette map.

[0021] Preferably, the input of the RGB personnel occupancy features and the RGB environment features into the slow feature analysis model for processing comprises the following steps.

[0022] The RGB personnel occupancy features and the RGB environment features are centralized, and a covariance matrix and a cross-covariance matrix corresponding to the RGB personnel occupancy features and the RGB environment features are calculated;

[0023] According to the covariance matrix and the cross-covariance matrix, a covariance matrix of first-order derivatives of input features and a covariance matrix of input features are calculated, a generalized eigenvector is obtained and normalized, and a mapping matrix is obtained;

[0024] According to the mapping matrix, the RGB personnel occupancy features and the RGB environment features, an RGB silhouette image is obtained.

[0025] Further, the RGB silhouette image is input into a space-time attention mechanism module for processing to obtain RGB space-time features, including:

[0026] According to the RGB silhouette image, a feature map and an attention map corresponding to the RGB silhouette image are obtained.

[0027] The feature map is divided into a plurality of spatial region feature maps, and the attention map is divided into a plurality of spatial attention maps;

[0028] The values of each spatial attention map are subjected to L1 normalization processing to obtain a spatial attention score of each region.

[0029] The spatial attention scores of the spatial attention maps at different time points in the same region are subjected to L1 normalization processing to obtain a space-time attention score.

[0030] According to the spatial region feature map and the space-time attention score, the RGB space-time features are obtained.

[0031] Further, the aggregation feature is obtained by fusing the RGB space-time features and the infrared features through a DFA operator.

[0032] Preferably, the fusion of the RGB space-time features and the infrared features through the DFA operator includes:

[0033] In each channel, the infrared space-time features are subjected to normalization processing using a Sigmoid activation function to obtain a normalized score.

[0034] The normalized parameters of the infrared space-time features are multiplied by the RGB space-time features to obtain a fused feature.

[0035] Further, the convolution processing of the aggregation feature to output an image containing a head target detection frame includes:

[0036] The aggregation feature is input into a first convolution layer for encoding processing, and then is input into a second convolution layer and a third convolution layer in parallel to obtain an image containing a target detection frame and a class.

[0037] The target detection frame is filtered by a non-maximum suppression algorithm, and an image containing a head target detection frame is output.

[0038] In a second aspect, the present application provides an indoor people counting system based on multi-element data fusion.

[0039] An indoor people counting system based on multi-element data fusion comprises:

[0040] An acquisition module is configured to acquire an RGB image and an infrared image, wherein the RGB image comprises an RGB indoor people occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor people occupancy map and an infrared indoor environment background map.

[0041] An indoor people counting module is configured to input the RGB image and the infrared image into a preset indoor people counting model for processing to obtain an indoor people count.

[0042] The RGB image and the infrared image are subjected to deep slow feature analysis to obtain an RGB silhouette map and an infrared silhouette map.

[0043] The RGB silhouette map and the infrared silhouette map are input into a space-time attention mechanism module for processing to obtain RGB space-time features and infrared space-time features.

[0044] The RGB space-time features and the infrared space-time features are used to obtain aggregated features.

[0045] The aggregated features are subjected to convolution processing to output an image containing a head target detection frame, the number of head target detection frames is calculated, and an indoor people count is obtained.

[0046] In a third aspect, the present application provides an electronic device.

[0047] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the steps of the above-mentioned indoor people counting method based on multi-element data fusion are completed.

[0048] In a fourth aspect, the present application provides a computer readable storage medium.

[0049] A computer readable storage medium is configured to store computer instructions, when the computer instructions are executed by a processor, the steps of the above-mentioned indoor people counting method based on multi-element data fusion are completed.

[0050] Compared with the prior art, the present application has the following beneficial effects:

[0051] 1. The technical scheme provided by the present application uses a multi-element information fusion method, fully utilizes the texture details and high spatial resolution information of the RGB image and the thermal radiation information of the infrared image, realizes the complementation of the two modal information, fuses the thermal radiation information while obtaining detailed texture information, combines the features of each dimension, enhances the feature expression, obtains more comprehensive features, solves the problem of scale change caused by the movement of the head of the person, and thus improves the detection accuracy.

[0052] 2. In order to highlight the difference between the indoor person occupancy graph and the indoor environment background graph, the technical scheme provided by the present application uses a deep slow feature analysis algorithm to remove the interference information contained in the indoor environment background graph from the current person occupancy graph, extracts features that can highlight the change of indoor personnel, enhances the features of the foreground region while weakening the background information, obtains an RGB silhouette graph and an infrared silhouette graph, and highlights the region of personnel change in the current person occupancy graph and the indoor environment background graph. To some extent, the complexity and congestion of small-scale scenes are alleviated, and the interference problem of background objects similar to the head information of the person is solved.

[0053] 3. The technical scheme provided by the present application inputs the RGB silhouette graph and the infrared silhouette graph output by the deep slow feature analysis into the space-time attention mechanism. The space-time attention mechanism first obtains the spatial correlation between the pixels of the RGB silhouette graph and the infrared silhouette graph through the spatial attention mechanism, and then captures the temporal correlation of the spatial features extracted from a plurality of RGB silhouette graphs and infrared silhouette graphs in a period of time through the temporal attention mechanism, to obtain RGB space-time features and infrared space-time features, thereby focusing on the space-time feature information of the RGB silhouette graph and the infrared silhouette graph, concentrating the space-time attention on the head information of the indoor personnel, quickly obtaining the most effective information, and improving the confidence of head detection.

[0054] 4. The technical scheme provided by the present application fuses the RGB space-time features and the infrared space-time features obtained according to the space-time attention mechanism using a DFA operator, constructs aggregated features, and then uses three layers of convolution to complete the encoding of the aggregated features and the output of the target head frame and the category. Finally, the number of target detection frames is calculated to complete the counting of indoor personnel. BRIEF DESCRIPTION OF DRAWINGS

[0055] The drawings accompanying the specification of the present application form part of the present application and serve to provide a further understanding of the present application, the exemplary embodiments of the present application and their description serve to explain the present application and do not constitute an improper limitation of the present application.

[0056] Figure 1 The flowchart provided for the embodiments of the present application;

[0057] Figure 2 The flowchart provided for the embodiments of the present application;

[0058] Figure 3 A flowchart of a personnel change region enhancement process based on deep slow feature analysis is provided for an embodiment of the present application.

[0059] Figure 4 A construction diagram of a variational autoencoder is provided for an embodiment of the present application.

[0060] Figure 5 A processing flowchart of a spatiotemporal attention mechanism is provided for an embodiment of the present application.

[0061] Figure 6 A flowchart of a human head recognition process is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] It should be noted that the following detailed description is merely exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0063] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0064] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0065] Embodiment One

[0066] The existing indoor personnel detection method based on head detection still has many limitations in a relatively complex indoor scene, which affects the accuracy of personnel detection. Therefore, the present application provides an indoor personnel counting method based on multi-element data fusion, which performs indoor personnel image target detection based on deep slow feature and multi-modal fusion.

[0067] Next, the present application will be described in detail in combination with Figures 1-6 A multi-element data fusion-based indoor personnel counting method disclosed in the present embodiment will be described in detail. The multi-element data fusion-based indoor personnel counting method comprises the following steps:

[0068] S1, acquire an RGB image and an infrared image, wherein the RGB image comprises an RGB indoor person occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor person occupancy map and an infrared indoor environment background map.

[0069] In the indoor person detection process, due to the complex indoor environment and the change of the person's posture, the feature information of the person in the image is easy to be blocked or the scale is changed due to the head movement. Therefore, in this embodiment, the infrared image data is fused on the basis of the RGB image data, the texture detail information of the RGB image and the thermal radiation information of the infrared image are fused to enhance the feature information of the person, and the detection accuracy of the person is improved.

[0070] Specifically, a synchronous system comprising an optical binocular camera and an infrared binocular camera is constructed in an indoor environment to capture double-mode images of the indoor environment at the same angle and at the same time. The optical binocular camera obtains an RGB indoor person occupancy map and an RGB indoor environment background map with texture details and high spatial resolution, and the infrared binocular camera obtains an infrared indoor person occupancy map with clear thermal radiation.

[0071] In this embodiment, the synchronous imaging system comprising the optical binocular camera and the infrared binocular camera is used to capture data, and n continuous indoor person occupancy maps in a period of time are selected for collection. The specific collection form is as shown in Figure 2 .

[0072] When training the indoor person counting model, image data is collected, and the Labelimg labeling tool is used to label the detection target (head) in the image with a corresponding label category (person).

[0073] Data labeling (or data annotation) is part of the preprocessing stage when developing a machine learning (ML) model, which is responsible for identifying raw data (such as images, text files, videos) and then adding one or more labels to the raw data to specify the context of the model, helping the machine learning model to make accurate predictions. This process ensures that each object detected in the training set is labeled.

[0074] S2, input the RGB image and the infrared image into a preset indoor person counting model for processing to obtain the number of indoor persons. The specific process of inputting the RGB image and the infrared image into the preset indoor person counting model for processing comprises:

[0075] S201, perform deep slow feature analysis on the RGB image and the infrared image to obtain an RGB silhouette map and an infrared silhouette map. The specific steps comprise:

[0076] S2011, input the RGB indoor person occupancy map and the RGB indoor environment background map in the RGB image dataset into the variational autoencoder respectively, obtain the RGB person occupancy feature and the RGB environment feature; input the infrared indoor person occupancy map and the infrared indoor environment background map in the infrared image dataset into the variational autoencoder respectively, obtain the infrared person occupancy feature and the infrared environment feature.

[0077] S2012, input the RGB person occupancy feature and the RGB environment feature into the slow feature analysis model for processing, obtain the RGB silhouette map; input the infrared person occupancy feature and the infrared environment feature into the slow feature analysis model for processing, obtain the infrared silhouette map.

[0078] The inputting of the RGB person occupancy feature and the RGB environment feature into the slow feature analysis model for processing includes the following steps:

[0079] (1) perform centering processing on the RGB person occupancy feature and the RGB environment feature, calculate the covariance matrix and the cross-covariance matrix of the corresponding matrix of the RGB person occupancy feature and the RGB environment feature after the centering processing.

[0080] (2) according to the covariance matrix and the cross-covariance matrix, calculate the covariance matrix of the first-order derivative of the input feature and the covariance matrix of the original input feature, solve the generalized eigenvector, and normalize to obtain a mapping matrix.

[0081] (3) according to the mapping matrix, the RGB person occupancy feature and the RGB environment feature, obtain the RGB silhouette map.

[0082] The inputting of the infrared person occupancy feature and the infrared environment feature into the slow feature analysis model for processing, to obtain the infrared silhouette map, includes the following steps:

[0083] (1) perform centering processing on the infrared person occupancy feature and the infrared environment feature, calculate the covariance matrix and the cross-covariance matrix of the corresponding matrix of the infrared person occupancy feature and the infrared environment feature after the centering processing.

[0084] (2) according to the covariance matrix and the cross-covariance matrix, calculate the covariance matrix of the first-order derivative of the input feature and the covariance matrix of the original input feature, solve the generalized eigenvector, and normalize to obtain a mapping matrix.

[0085] (3) according to the mapping matrix, the infrared person occupancy feature and the infrared environment feature, obtain the infrared silhouette map.

[0086] Next, combined with Figure 3 The process of S201 is described in detail taking the RGB image as an example.

[0087] The RGB indoor occupancy map X and the RGB indoor environment background map Y are input into the variational autoencoder VAE in the form of pixels X, Y ∈ R m×n , where R m×n represents a matrix, and m and n represent the number of feature bands and the number of pixels, respectively. The model structure of the VAE includes two parts, a variational autoencoder and a decoder. The features on the picture are obtained through the variational autoencoder to obtain the latent vector Z, and then the decoder reconstructs the original input image X' and Y' according to the features. In the training process, the goal of the VAE is to minimize the reconstruction error between the input X and X'.

[0088] As shown in Figure 4 , X is the data that can be observed, which represents the RGB indoor occupancy map in this case. Z is the latent variable, that is, the features on the RGB indoor occupancy map obtained through the variational autoencoder X' is the reconstructed data, which represents the reconstructed RGB indoor occupancy map. The overall execution process of the VAE is as follows:

[0089] First, the encoder g encodes the RGB indoor occupancy map into the latent variable distribution parameter, that is:

[0090]

[0091] where μ is the mean and σ is the variance.

[0092] Then, the latent variable Z is sampled from the latent variable parameter to obtain the features on the RGB indoor occupancy map , that is:

[0093]

[0094] The decoder f reconstructs the original data from the latent variable Z to obtain the reconstructed data X', that is:

[0095]

[0096] where θ2 is the decoder parameter. The reconstruction error between the input X and X' can be measured by the KL divergence between the two distributions.

[0097] Finally, the features on the RGB indoor occupancy map X output by the variational autoencoder are obtained Similarly, the features of the RGB indoor environment background map Y are extracted by the above method Through the variational autoencoding network, the original RGB indoor occupancy map X and the RGB indoor environment background map Y are mapped to a new high-dimensional feature space, and their respective features and

[0098] Features extracted by variational autoencoder and Input slow feature analysis model.

[0099] First, analyze the input data. and Centralized processing is performed to obtain and Where, 1∈R T×T It is a matrix with all elements being 1.

[0100]

[0101] Calculate matrix and Covariance matrix and cross-covariance matrix:

[0102]

[0103] Where I is the identity matrix and r is the regularization constant.

[0104]

[0105] The objective function of SFA can be expressed as:

[0106]

[0107] This optimization problem is transformed into a problem of solving generalized eigenvalues:

[0108] Αω j =Bω j λ j (2-8)

[0109] Where, λ j It is the generalized eigenvalue corresponding to the j-th generalized eigenvector. When the generalized eigenvector ω is solved... j By using ω j Normalization yields the final projection vector.

[0110]

[0111] Further construct the projection matrix in This indicates the relationship between the first q smallest eigenvalues ​​λ. j The corresponding feature vector.

[0112] Features extracted by variational autoencoder network and Analysis based on slow features:

[0113]

[0114] The final RGB silhouette image is obtained. Similarly, the deep slow feature method is applied to the infrared image dataset to obtain an infrared silhouette image. The role of using SFA constraints to weaken the background image information and highlight the change component of the personnel information is achieved.

[0115] The indoor personnel occupancy map is an image containing person information and background information, the indoor environment background image is an image containing only background information, and the silhouette image is a silhouette obtained by cutting the indoor personnel occupancy map from the indoor environment background image, which cuts off the invariant background information.

[0116] S202, input the RGB silhouette image and the infrared silhouette image into the space-time attention mechanism module for processing to obtain RGB space-time features and infrared space-time features. The specific steps include:

[0117] S2021, according to the RGB silhouette image, obtain the corresponding feature map and attention map; according to the infrared silhouette image, obtain the corresponding infrared feature map and infrared attention map.

[0118] S2022, divide the feature map into multiple, obtain multiple spatial region feature maps; divide the attention map into multiple, obtain multiple spatial attention maps; divide the infrared feature map into multiple, obtain multiple infrared spatial region feature maps; divide the infrared attention map into multiple, obtain multiple infrared spatial attention maps.

[0119] S2023, perform L1 normalization processing on all values of each spatial attention map to obtain the spatial attention score of each region; perform L1 normalization processing on the spatial attention scores of the spatial attention maps at different time points in the same region to obtain the space-time attention score. Perform L1 normalization processing on all values of each infrared spatial attention map to obtain the spatial attention score of each region; perform L1 normalization processing on the spatial attention scores of the infrared spatial attention maps at different time points in the same region to obtain the space-time attention score.

[0120] S2024, according to the spatial region feature map and the corresponding space-time attention score, obtain the RGB space-time feature; according to the infrared spatial region feature map and the corresponding space-time attention score, obtain the infrared space-time feature.

[0121] The space-time attention mechanism focusing on the head information of the personnel is added to the pre-trained model ResNet50 architecture, which focuses on the head information of the personnel. This module uses the space-time attention mechanism to automatically assign attention weights containing spatial attention information and temporal attention information to each spatial region in different images. The input of the space-time attention mechanism is the RGB silhouette image dataset and the infrared silhouette image dataset processed by the deep slow feature analysis. Taking the RGB silhouette image dataset as an example, the working principle of the space-time attention mechanism is as follows:

[0122] The RGB silhouette image data processed by the deep slow feature is input into a convolutional layer to obtain a feature map {f n} {n＝1:N} Then, the sum of squares of the depth channel is L2 normalized to generate a corresponding attention map g n , as shown in equation (3-1):

[0123]

[0124] where h and w are the height and width of the feature map. Let G be the set of attention maps generated by the input silhouette images, G = [g1, g2,... g n ], and suppose g i and g j are the attention maps calculated by (3-1), the square of the Frobenius norm of the difference between g i and g j is used, as shown in equation (3-2):

[0125]

[0126] where g i and g j are the attention maps generated by two randomly selected silhouette images. This regularization term Reg is multiplied by a coefficient and added to the loss function to minimize it, completing the regularization.

[0127] Each RGB silhouette image has a corresponding attention map, and the feature maps and attention maps of n RGB silhouette images are divided into K blocks, as shown in equation (3-3):

[0128]

[0129] where g n,k represents the spatial attention map of the kth region of the nth RGB silhouette image. L1 normalization is performed on all values of each block to obtain a spatial attention score for the region:

[0130]

[0131] L1 normalization is performed on the spatial attention scores of the RGB silhouette images at different times in the same region to obtain normalized spatiotemporal attention scores, and each spatial region from a different RGB silhouette image is assigned a specific spatiotemporal attention score S(n, k):

[0132]

[0133] An N*K matrix S is obtained through inter-regularization of RGB silhouette maps, which is a feature map f of each spatial region of each RGB silhouette map n,k An attention score S(n, k) is assigned. Each attention score is used as a weight, and matrix multiplication is performed on each segmented feature map element by element to generate another RGB space-time feature F1 containing global information of the input image:

[0134] F l = f nk ×s nk (3-6)

[0135] Similarly, after the infrared silhouette map data passes through the feature extraction network containing the space-time attention mechanism, the infrared space-time feature F2 can also be obtained.

[0136] F2 = f nk ×s nk (3-7)

[0137] S203, according to the RGB space-time feature and the infrared space-time feature, obtaining the aggregated feature; performing convolution processing on the aggregated feature, outputting an image containing a head target detection frame, calculating the number of head target detection frames, and obtaining the number of indoor personnel. Specifically, it includes:

[0138] S2031, fuse the RGB space-time feature and the infrared feature by DFA operator to obtain the aggregated feature.

[0139] Specifically, the infrared space-time feature is normalized by using a Sigmoid activation function in each channel to obtain a normalized score; the normalized parameter of the infrared space-time feature is multiplied by the RGB space-time feature, and the features reduced after the normalization of the infrared space-time feature are supplemented to obtain the fused feature.

[0140] S2032, input the aggregated feature into the first convolutional layer for encoding processing, and then input the second convolutional layer and the third convolutional layer in parallel to obtain an image containing a target detection frame and a category.

[0141] S2033, filter the target detection frame by a non-maximum suppression algorithm, output an image containing a head target detection frame, calculate the number of head target detection frames, and determine the number of indoor personnel.

[0142] In this step, the complementary information of visible light and infrared is fully utilized to fuse the features extracted from the RGB image and the infrared image to obtain an aggregated feature h agg , and then the aggregated feature h agg is decoded by a deep feature aggregation DFA operator to output a head detection frame and a category.

[0143] For example,Figure 6 As shown, the input of the module is the RGB spatio-temporal feature F1 and the infrared spatio-temporal feature F2 obtained by the spatio-temporal attention mechanism. In order to utilize the complementary information of visible light and infrared, the features extracted from the RGB image and the infrared image are fused to obtain the aggregated feature h agg The specific steps are as follows:

[0144] Firstly, the deep feature aggregation DFA operator is used to fuse the RGB and infrared spatio-temporal features, enhance the personnel information instances and weaken the background information, and the DFA operator is shown in equation (4-1):

[0145] DFA: h agg = a · F1☉σ(F2) + b · F2 (4-1)

[0146] The DFA operator first normalizes the infrared spatio-temporal feature F2 to [0, 1] in each channel using the Sigmoid activation function, and the formula of the Sigmoid activation function is shown in equation (4-2):

[0147]

[0148] Then, the score of the normalized infrared spatio-temporal feature is multiplied with the corresponding element of the RGB spatio-temporal feature value F1 in the corresponding tensor, and the Hadamard product is shown in equation (4-3):

[0149]

[0150] Wherein, σ(F2) can be regarded as a special mask to activate the local information of the original feature map F1, and the indoor personnel information is retained and the static background interference information is weakened. The infrared spatio-temporal feature F2 is normalized to [0, 1], and the RGB spatio-temporal feature F1 is normalized before being activated, and the activated feature will be reduced. In order to compensate for this loss and enhance the area of the head feature in the infrared image, F2 is added after the element-level multiplication of the equation.

[0151] After obtaining the fused feature, three convolutional layers are established before the regression coordinates and classification, the first convolutional layer is to encode the information obtained from the previous layer, the second convolutional layer is the Regression head responsible for predicting the positioning coordinates, and multiple detection frames are generated according to the target positioning; the third convolutional layer is the Classification head responsible for classification and prediction of the probability score of the head of the personnel.

[0152] Different from image classification which has only one output, the output number of object detection is unknown. In the post-processing stage of object detection, most of the candidate bounding boxes with confidence lower than a threshold can be filtered out. For the same object, there may be multiple bounding boxes with confidence higher than the threshold, so the non-maximum suppression (NMS) algorithm is needed to remove the repeated bounding boxes.

[0153] The intersection over union (IOU) is the basis for object detection NMS, and the position of the bounding box is measured by the IOU index. The calculation of the overlap between the real bounding box and the predicted bounding box is shown in (4-4):

[0154]

[0155] The standard NMS suppression function is shown in (4-5):

[0156]

[0157] where S i represents the confidence of each bounding box, M is the current bounding box with the highest confidence, b i is one of the remaining boxes, N t is a set threshold, and when the intersection over union IOU is greater than N t , the score of the bounding box is 0, and the image is directly discarded.

[0158] Finally, the number of remaining head detection boxes is calculated to obtain the number of indoor personnel.

[0159] Embodiment Two

[0160] The embodiment discloses an indoor personnel counting system based on multi-element data fusion, comprising:

[0161] An acquisition module is configured to acquire an RGB image and an infrared image, wherein the RGB image comprises an RGB indoor personnel occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor personnel occupancy map and an infrared indoor environment background map.

[0162] An indoor personnel counting module is configured to input the RGB image and the infrared image into a preset indoor personnel counting model for processing to obtain the number of indoor personnel; wherein the inputting the RGB image and the infrared image into the trained indoor personnel counting model for processing comprises:

[0163] performing deep slow feature analysis on the RGB image and the infrared image to obtain an RGB silhouette map and an infrared silhouette map;

[0164] The RGB silhouette image and the infrared silhouette image are input into a space-time attention mechanism module respectively for processing to obtain RGB space-time features and infrared space-time features.

[0165] According to the RGB space-time features and the infrared space-time features, aggregated features are obtained.

[0166] The aggregated features are subjected to convolution processing to output an image containing a head target bounding box, the number of head target bounding boxes is calculated, and the number of indoor personnel is obtained.

[0167] It should be noted that the above obtaining module and indoor personnel counting module correspond to the steps in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0168] Embodiment Three

[0169] Embodiment Three of the present application provides an electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are executed by the processor, the steps of the above-mentioned indoor personnel counting method based on multi-element data fusion are completed.

[0170] Embodiment Four

[0171] Embodiment Four of the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps of the above-mentioned indoor personnel counting method based on multi-element data fusion are completed.

[0172] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0173] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0174] These computer program instructions can also be loaded into computer or other programmable data processing devices, to cause a series of operational steps to be performed on the computer or other programmable devices, so that the computer program instructions which execute on the computer or other programmable devices provide steps for implementing the function specified in the flowchart Figure 1 one or more processes and / or blocks Figure 1 the function specified in the one or more blocks.

[0175] The above description of the various embodiments has focused on individual embodiments, and the parts not described in detail in a certain embodiment can be seen from the relevant description of other embodiments.

[0176] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for indoor people counting based on multi-element data fusion, characterized in that, The method comprises the following steps: obtaining an RGB image and an infrared image, wherein the RGB image comprises an RGB indoor person occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor person occupancy map and an infrared indoor environment background map; inputting the RGB image and the infrared image into a preset indoor person counting model for processing to obtain the number of indoor persons; wherein the inputting the RGB image and the infrared image into the trained indoor person counting model for processing comprises: performing deep slow feature analysis on the RGB image and the infrared image to obtain an RGB silhouette map and an infrared silhouette map, comprising: inputting the RGB indoor person occupancy map and the RGB indoor environment background map into a variational autoencoder respectively to obtain an RGB person occupancy feature and an RGB environment feature; inputting the RGB person occupancy feature and the RGB environment feature into a slow feature analysis model for processing to obtain the RGB silhouette map, comprising: performing centering processing on the RGB person occupancy feature and the RGB environment feature, and calculating a corresponding covariance matrix and a cross-covariance matrix; calculating the covariance matrix of the first-order derivative of the input feature and the covariance matrix of the input feature according to the covariance matrix and the cross-covariance matrix, obtaining a generalized eigenvector and performing normalization processing to obtain a mapping matrix; obtaining the RGB silhouette map according to the mapping matrix, the RGB person occupancy feature and the RGB environment feature; inputting the RGB silhouette map and the infrared silhouette map into a spatio-temporal attention mechanism module for processing to obtain an RGB spatio-temporal feature and an infrared spatio-temporal feature; obtaining an aggregated feature according to the RGB spatio-temporal feature and the infrared spatio-temporal feature; performing convolution processing on the aggregated feature to output an image containing a head target detection frame, calculating the number of head target detection frames, and obtaining the number of indoor persons.

2. The multi-data fusion based indoor people counting method of claim 1, wherein, The processing of the RGB silhouette map in the spatio-temporal attention mechanism module to obtain the RGB spatio-temporal feature comprises: obtaining a corresponding feature map and an attention map according to the RGB silhouette map; dividing the feature map into multiple to obtain multiple spatial region feature maps; and dividing the attention map into multiple to obtain multiple spatial attention maps; performing L1 normalization processing on all values of each spatial attention map to obtain a spatial attention score of each region; performing L1 normalization processing on the spatial attention scores of the spatial attention maps at different time points in the same region to obtain a spatio-temporal attention score; obtaining the RGB spatio-temporal feature according to the spatial region feature map and the spatio-temporal attention score.

3. The multi-data fusion based indoor people counting method of claim 1, wherein, The obtaining of the aggregated feature according to the RGB spatio-temporal feature and the infrared spatio-temporal feature specifically comprises: fusing the RGB spatio-temporal feature and the infrared feature through a DFA operator to obtain the aggregated feature.

4. The method for indoor people counting based on multi-data fusion according to claim 3, wherein, The fusion of the RGB spatio-temporal feature and the infrared feature through the DFA operator comprises: normalizing the infrared spatio-temporal feature in each channel using a Sigmoid activation function to obtain a normalization score; performing a product operation on the normalization parameter of the infrared spatio-temporal feature and the RGB spatio-temporal feature to obtain a fused feature.

5. The multi-data fusion based indoor people counting method of claim 1, wherein, The convolution processing of the aggregated feature to output an image containing a head target detection frame comprises: The polymeric features are input into a first convolutional layer for encoding processing, and then are input into a second convolutional layer and a third convolutional layer in parallel to obtain an image containing a target detection frame and a category; The target detection frame is filtered through a non-maximum suppression algorithm to output an image containing a head target detection frame.

6. An indoor people counting system based on multi-element data fusion, characterized in that, The method comprises the steps of: An acquisition module is configured to acquire an RGB image and an infrared image, wherein the RGB image comprises an RGB indoor person occupancy map and an RGB indoor environment background map, and the infrared image comprises an infrared indoor person occupancy map and an infrared indoor environment background map; An indoor person counting module is configured to input the RGB image and the infrared image into a preset indoor person counting model for processing to obtain an indoor person count; wherein the inputting of the RGB image and the infrared image into the trained indoor person counting model for processing comprises the steps of: performing deep slow feature analysis on the RGB image and the infrared image to obtain an RGB silhouette map and an infrared silhouette map, including inputting the RGB indoor person occupancy map and the RGB indoor environment background map into a variational autoencoder respectively to obtain an RGB person occupancy feature and an RGB environment feature; inputting the RGB person occupancy feature and the RGB environment feature into a slow feature analysis model for processing to obtain the RGB silhouette map, including performing centering processing on the RGB person occupancy feature and the RGB environment feature, and calculating a corresponding covariance matrix and a cross-covariance matrix; calculating a covariance matrix of a first-order derivative of the input feature and a covariance matrix of the input feature according to the covariance matrix and the cross-covariance matrix, obtaining a generalized feature vector and performing normalization processing to obtain a mapping matrix; obtaining the RGB silhouette map according to the mapping matrix, the RGB person occupancy feature and the RGB environment feature; inputting the RGB silhouette map and the infrared silhouette map into a spatio-temporal attention mechanism module for processing to obtain an RGB spatio-temporal feature and an infrared spatio-temporal feature; obtaining aggregated features according to the RGB spatio-temporal feature and the infrared spatio-temporal feature; performing convolution processing on the aggregated features to output an image containing a head target detection frame, calculating the number of head target detection frames, and obtaining an indoor person count.

7. An electronic device, comprising: The computer instructions are executed by the processor to complete the steps of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer instructions are executed by the processor to complete the steps of any one of claims 1-5.

Images