Personnel flow density statistics and alarm processing method and system based on federated learning

Abstract

The application discloses a personnel flow density statistics and alarm processing method and system based on federated learning. The FairMOT algorithm is used for personnel identification on the image data set in a single camera, and then the image recognition results in multiple cameras are sequentially associated in time, and finally the personnel flow density in the entire property project monitoring range is counted, and the area with high personnel flow density in the property project is alarmed in a process, so as to achieve the purposes of epidemic prevention and control and improvement of property service quality. Compared with the prior art, the application fuses the federated learning algorithm and the FairMOT algorithm to build an algorithm platform, realizes multi-camera multi-target deduplication tracking, and the algorithm not only has the distributed advantage, but also can fuse multiple local edge models to form a cloud global model of the entire property project under the condition that data does not go out of the local edge, can effectively protect the data privacy of the owners, break through the data barrier, and realize the centralized allocation of computing capacity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a personnel flow density statistical and alarm processing method and system based on federated learning. BACKGROUND

[0002] The residential community and the entrance and exit section of the commercial park, the school nearby section, the public transportation stop section, the leisure green section, the crowd gathers, the property company can statistically, monitor the personnel flow peak of each community in real time through the algorithm technology, and remind or personnel evacuation in the field when necessary.

[0003] Therefore, it is necessary to provide a personnel flow density statistical and alarm processing method and system, which regularly counts the personnel flow density in a residential community or a commercial park, and alarm the area with too high personnel flow density through the construction of emergency strategy, so as to improve the quality of property service. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a personnel flow density statistical and alarm processing method and system based on federated learning, which aims to build an algorithm platform that can effectively protect the data privacy of the owners and can intensively allocate the system computing power, alarm the area with too high personnel flow density in the property project in a process, and improve the quality of property service.

[0005] To solve the above technical problems, the present application adopts the following technical solutions:

[0006] A personnel flow density statistical and alarm processing method based on federated learning, comprising the following steps:

[0007] S10, a plurality of local edge ends utilize the monitored video data acquired by each to locally train the FairMOT network to acquire a local edge end model; S20, the cloud server aggregates the change in the backbone network parameters of the local edge end model acquired from the local edge ends, and updates the cloud global model using the change in the backbone network parameters of the local edge end model after the aggregation; S30, each local edge end downloads the change in the parameters of the cloud global model from the cloud server, combines the classifier of the local edge end model, and utilizes the monitored video data acquired by each to perform a new round of local training; S40, steps S20 to S30 are repeatedly executed until the cloud global model meets the convergence requirement, and then the training of the cloud global model and the local edge end model is completed; S50, each local edge end utilizes the trained local edge end model to perform personnel identification and personnel flow density statistics on the real-time monitored video data acquired by each, and outputs the personnel flow density statistics result; S60, the personnel flow density statistics result is matched with a preset people flow density alarm strategy, and when the personnel flow density reaches an alarm trigger point, an alarm work order is generated.

[0008] A personnel flow density statistics and alarm processing system based on federated learning includes a cloud server and a plurality of local edge ends. The cloud server includes a global model training module and a training termination judgment module. Each local edge end includes a local model training module, a people flow density statistics module, and an alarm module. The global model training module is configured to aggregate the change in the backbone network parameters of the local edge end model acquired from the local edge ends, and update the cloud global model using the change in the backbone network parameters of the local edge end model after the aggregation. The local model training module is configured to utilize monitored video data to locally train the FairMOT network to acquire a local edge end model, and to download the change in the parameters of the cloud global model from the cloud server, combine the classifier of the local edge end model, and utilize monitored video data to perform a new round of local training. The training termination judgment module is configured to judge whether the cloud global model meets the convergence requirement, and to determine whether to terminate the training of the cloud global model and the local edge end model according to the judgment result. The people flow density statistics module is configured to utilize the trained local edge end model to perform personnel identification and personnel flow density statistics on real-time monitored video data, and to output the personnel flow density statistics result. The alarm module is configured to match the personnel flow density statistics result with a preset people flow density alarm strategy, and to generate an alarm work order when the personnel flow density reaches an alarm trigger point.

[0009] The beneficial technical effect of the present application is that the present application uses the FairMOT algorithm to identify personnel from the image data set extracted in a single camera device, then time-sequentially associates the image recognition results of multiple camera devices, and finally counts the pedestrian density in the entire property project monitoring range, and alarms the process of the area with high personnel flow density in the property project, so as to improve the quality of property services. Compared with the prior art, the present application builds an algorithm platform by fusing the federated learning algorithm and the FairMOT algorithm, realizes multi-camera multi-target deduplication tracking, and the algorithm not only has the advantage of distribution, but also can form a cloud global model of the entire property project by fusing multiple local edge models under the condition that data does not go out of the local edge, can effectively protect the data privacy of the owners, break through the data barrier, and realize the centralized allocation of computing capacity. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 The flowchart of the personnel flow density statistical and alarm processing method based on federated learning of the present application is shown in the figure.

[0011] Figure 2 The personnel flow density statistical sub-flowchart of the present application is shown in the figure.

[0012] Figure 3 The Re-ID re-identification tracking sub-flowchart of the present application is shown in the figure.

[0013] Figure 4 The personnel flow density alarm sub-flowchart of the present application is shown in the figure. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to more clearly understand the purpose, technical scheme and advantages of the present application, the present application will be further described below in conjunction with the drawings and examples.

[0015] As shown in the figure, in one embodiment of the present application, the personnel flow density statistical and alarm processing method based on federated learning includes steps S10 to S60: Figure 1

[0016] S10, multiple local edges use the monitoring video data obtained by themselves to train the FairMOT network locally to obtain a local edge model.

[0017] ​A plurality of cameras are uniformly deployed in the public area of a property project (residential community or commercial garden), and real-time capture of personnel gathering in different positions is performed. The monitoring video data collected by each camera is used as a local data set. Each local edge obtains monitoring video data through a corresponding camera, and after frame extraction and size adjustment processing of the monitoring video data, image data meeting the model input requirements is obtained, and the image data is input into the FairMOT network for local training to obtain a local edge model.

[0018] Generally, the federated learning algorithm requires full synchronization of model parameters between the local edge and the cloud, but for the re-identification algorithm, the dimension of the classification layer is determined by the number of data set samples used by each local edge. Therefore, in order to complete the cloud global model training, the present embodiment uses a partially synchronized federated learning algorithm under the FairMOT algorithm Re-ID re-identification branch. After local training, the backbone network parameters and the classifier are separated, only the local edge model backbone network parameter changes are homomorphically encrypted and uploaded to the cloud server, and the classifier of the local edge model is always updated in the local edge. The update method is as follows:

[0019]

[0020] wherein, represents the local edge model of the s-th round, represents the classifier of the local edge model of the s-th round, and η represents the learning rate, b) represents the gradient calculated by the local data b and the current local edge model classifier .

[0021] S20, the cloud server obtains the local edge model backbone network parameter changes from the local edge for aggregation processing, and updates the cloud global model using the aggregated local edge model backbone network parameter changes.

[0022] While the local edge model is trained in the local edge, the cloud server randomly samples m local edges, obtains the local edge model backbone network parameter changes obtained by the m local edges using private data for training, and uploads the local edge model backbone network parameter changes homomorphically encrypted by the local edge to the cloud server, wherein m is less than or equal to the total number of local edges; the cloud server decrypts and performs weighted average calculation on the m local edge model backbone network parameter changes to generate a new round of global model parameters; the cloud server updates the cloud global model using the new round of global model parameters:

[0023]

[0024] wherein, denotes the cloud global model of the s+1th round, denotes the local edge model of the s+1th round, n m is the data volume of the local edge model sampled in the current round of training, n all denotes the total data volume of all local edge models.

[0025] S30, each local edge downloads the parameter change amount of the cloud global model from the cloud server, combines the classifier of the local edge model, and uses the monitored video data obtained by itself to perform a new round of local training.

[0026] A plurality of local edges obtain respective monitored video data through corresponding cameras, and take the respective monitored video data obtained by themselves as a local data set. Then, the parameter change amount of the cloud global model is downloaded from the cloud server, the classifier of the local edge model is combined, and the local data set (private data) is used to perform a new round of local training. After the local training is completed, the parameter change amount of the backbone network of the local edge model is homomorphically encrypted and uploaded to the cloud server, and the classifier of the local edge model is left in the local edge for updating.

[0027] S40, repeat steps S20 to S30 until the cloud global model meets the convergence requirement, then complete the training of the cloud global model and the local edge model.

[0028] Steps S20 to S30 are repeatedly performed between the cloud server and the local edge. Generally, the cloud global model can meet the convergence requirement after being repeated more than 4 times, at which time the training of the cloud global model and the local edge model is terminated.

[0029] S50, each local edge uses the trained local edge model to perform personnel recognition and personnel flow density statistics on the real-time monitored video data obtained by itself, and outputs the personnel flow density statistics result.

[0030] As shown in Figure 2 , the step S50 further includes steps S51 to S54:

[0031] S51, obtain real-time monitored video data and perform preprocessing to generate real-time monitoring images meeting the model input requirements.

[0032] Specifically, the local edge obtains real-time monitored video data through a camera and performs preprocessing. After frame extraction, size adjustment processing is performed on the real-time monitored video data, real-time monitoring images meeting the model input requirements are obtained, and data deduplication, enhancement, filtering and other processing are performed on the real-time image data.

[0033] S52, identify the time of the real-time monitoring image using an OCR algorithm, and determine the real-time monitoring image within the statistical range of the current rolling time window according to the time of the real-time monitoring image, denoted as a to-be-tested monitoring image.

[0034] This step applies a rolling time window statistical method. The time of the real-time monitoring image is detected using an OCR algorithm, and the identified time of the real-time monitoring image is matched with the time of the current rolling time window, so as to determine the real-time monitoring image within the statistical range of the current rolling time window, denoted as a to-be-tested monitoring image.

[0035] S53, performing personnel identification and personnel flow density statistics on the to-be-tested monitoring image using the trained local edge model, and outputting the personnel flow density statistical result.

[0036] The step S53 further comprises steps S531 to S534:

[0037] S531, inputting the to-be-tested monitoring image into the backbone network of the local edge model for feature extraction, and outputting the feature maps of 4 branches.

[0038] The local edge inputs the to-be-tested monitoring image into the DLA-34 network of the local edge model for feature extraction, and outputs the feature maps of 4 branches, wherein the size of the input to-be-tested monitoring image is represented as H image ×W image , and the shape of the output feature map is H×W×C, and (H, W) is the pixel of the to-be-tested monitoring image, and C is the number of channels of the to-be-tested monitoring image.

[0039] S532, inputting the feature maps of 3 branches into the 3 parallel heads of the multi-parallel head module of the target detection network of the local edge model, and the 3 parallel heads respectively output the center position, recognition frame size and center offset of each detection object.

[0040] This step inputs the feature maps of 3 branches generated by the backbone network into the target detection branch network (CenterNet) in the FairMOT model, and uses the multi-parallel head module to perform the following 3 target estimation tasks. This method uses an anchor-free method, and three parallel heads are attached to the DLA-34 network to respectively output the center position, recognition frame size and center offset of each detection object. Each parallel head is realized by applying a 3×3 convolution to the output features of the DLA-34 network, and then a 1×1 convolution layer is finally generated.

[0041] Calculate the detection center position: for each detection frame in the image, use GT box to represent, and take the coordinates of the opposite corner points of the rectangular detection frame as a set Calculate the average center point Again, the corresponding position in the backbone network output feature map is calculated by the formula And Then, the distance between the position (x, y) in the heat map and the object center is calculated by the heat map response formula The distance between the position (x, y) in the heat map and the object center is calculated by the heat map response formula The response is exponentially attenuated. Where N represents the number of objects in the image, σ c represents the standard deviation.

[0042] Calculate the bounding box size: the height and width of the target bounding box of each detection box can be calculated as The accuracy will affect the evaluation of object detection performance.

[0043] Calculate the center position offset: the GT box offset is

[0044] S533, input the feature map of one branch into the Re-ID image reclassification branch structure in the local edge model, and perform object Re-ID feature extraction for personnel Re-ID tracking.

[0045] This step applies a 128xHxW feature map, which can extract the distinguishing features of personnel at each location.

[0046] As shown in Figure 3 , first, the feature map is feature-extracted using a residual network to obtain a feature vector. The principle formula of the residual network is y l = h(x l ) + F(x l , W l ) and x l+1 = f(y l ), x l is the residual block of the lth layer, h(x l ) is a direct mapping, and f(y l ) is a ReLU activation function.

[0047] Next, the feature vector is converted into one-hot encoding that can construct a loss function through one or more fully connected layers.

[0048] Then, a loss function is constructed using the cross-entropy method The entropy formula is The KL divergence formula is Since the data set pictures are collected from different geographical locations and different angles, they are heterogeneous, so represents the feature corresponding to the sampling data in the ith camera device, ​This represents the feature corresponding to the sampled data in the j-th camera device, and the two are different.

[0049] The next step is to calculate the metric matrix using cosine distance and test the distance between eigenvectors to evaluate the performance of the local model. After normalizing the eigenvectors, the distance between features is calculated using cosine distance.

[0050] The final step involves using the K-nearest neighbors method to select the K most closely matched images, resulting in a ranked list for evaluating the accuracy of the local model. Image matching is performed using the K-nearest neighbors method. If two similar images A and B are present, B should be among A's K nearest neighbors, and vice versa. However, if two dissimilar images C and D are present, even if C is among D's K nearest neighbors, D will not be among C's K nearest neighbors. Given an image p and an image set g, the distance between image p and all images in dataset g needs to be calculated, involving the formula... R(p,k)={(g i ∈N(p,k))∩(p∈N(g i Let N(p,k) be the set of images that satisfy the K-nearest neighbors, where k = N(p,k)| = |{g0,g1,g3,…,g...} k}|.

[0051] S54. When the current rolling time window ends, end the current round of statistics, jump to step S51, and start a new round of statistics.

[0052] S60. Match the statistical results of personnel flow density with the preset personnel flow density alarm strategy. When the personnel flow density reaches the alarm trigger point, generate an alarm work order.

[0053] like Figure 4 As shown, step S60 further includes the following steps: S61, obtaining the statistical results of personnel flow density; S62, determining whether the personnel flow density has reached the alarm threshold, and generating an alarm work order when the personnel flow density reaches the alarm threshold.

[0054] based on Figure 1 The embodiment shown illustrates a federated learning-based method for people flow density statistics and alarm processing. This invention provides a federated learning-based system for people flow density statistics and alarm processing, which includes a cloud server and multiple local edge devices. The cloud server includes a global model training module and a training termination judgment module, and each local edge device includes a local model training module, a people flow density statistics module, and an alarm module.

[0055] The global model training module is configured to obtain the change amount of the backbone network parameters of the local edge-end model from the local edge-end, perform aggregation processing on the change amount of the backbone network parameters of the local edge-end model, and update the cloud-side global model by using the change amount of the backbone network parameters of the local edge-end model after the aggregation processing;

[0056] The local model training module is configured to perform local training on the FairMOT network by using the monitoring video data to obtain the local edge-end model, and to download the change amount of the parameters of the cloud-side global model from the cloud-side server, combine the classifier of the local edge-end model, and perform a new round of local training by using the monitoring video data;

[0057] The training termination judgment module is configured to judge whether the cloud-side global model meets the convergence requirement, and judge whether to terminate the training of the cloud-side global model and the local edge-end model according to the judgment result.

[0058] The crowd density statistical module is configured to perform personnel identification and personnel flow density statistics on the real-time monitoring video data by using the trained local edge-end model, and output the personnel flow density statistical result.

[0059] The alarm module is configured to match the personnel flow density statistical result with a preset crowd density alarm strategy, and generate an alarm work order when the personnel flow density reaches an alarm triggering point.

[0060] The above description is merely preferred embodiments of the present application, and is not intended to limit the present application in any form. Those skilled in the art can make various equivalent changes and improvements on the basis of the above-described embodiments, and any equivalent changes or modifications made within the scope of the claims shall fall within the protection scope of the present application.

Claims

1. A method for people flow density statistics and alarm processing based on federated learning, characterized in that, The personnel flow density statistics and alarm processing method based on federated learning comprises the following steps: S10, multiple local edge ends use the monitored video data obtained by each local edge end to locally train the FairMOT network to obtain a local edge end model; S20, the cloud server aggregates the backbone network parameter change amount of the local edge end model obtained from the local edge end, and updates the cloud global model using the aggregated backbone network parameter change amount of the local edge end model; S30, each local edge end downloads the parameter change amount of the cloud global model from the cloud server, combines the classifier of the local edge end model, and uses the monitored video data obtained by each local edge end to perform a new round of local training; S40, steps S20 to S30 are repeatedly executed until the cloud global model meets the convergence requirement, and then the training of the cloud global model and the local edge end model is completed; S50, each local edge end uses the trained local edge end model to perform personnel identification and personnel flow density statistics on the real-time monitored video data obtained by each local edge end, and outputs the personnel flow density statistics result; S60, the personnel flow density statistics result is matched with a preset personnel flow density alarm strategy, and when the personnel flow density reaches an alarm triggering point, an alarm work order is generated; The step S50 further comprises the following steps: S51, real-time monitored video data is obtained and preprocessed to generate real-time monitoring images meeting the model input requirements; S52, an OCR algorithm is used to identify the time of the real-time monitoring image, and the real-time monitoring image belonging to the statistical range of the current rolling time window is determined according to the time of the real-time monitoring image, which is recorded as a to-be-tested monitoring image; S53, the trained local edge end model is used to perform personnel identification and personnel flow density statistics on the to-be-tested monitoring image, and the personnel flow density statistics result is outputted; S54, when the end time of the current rolling time window is reached, the current round of statistics is ended, and step S51 is jumped to start a new round of statistics; The step S53 further comprises the following steps: S531, the to-be-tested monitoring image is inputted into the backbone network of the local edge end model for feature extraction, and four branch feature maps are outputted; S532, three branch feature maps are inputted into three parallel heads of the multi-parallel head module of the target detection network of the local edge end model, and the three parallel heads respectively output the center position, the recognition frame size and the center offset of each detection object; S533, one branch feature map is inputted into the Re-ID image re-classification branch structure in the local edge end model for object Re-ID feature extraction to perform personnel re-identification tracking. 2.The personnel flow density statistics and alarm processing method based on federated learning according to claim 1, wherein, The step S20 further comprises the following steps: S21, the cloud server randomly samples m local edge ends to obtain the backbone network parameter change amount of the m local edge end models, wherein the backbone network parameter change amount of the local edge end model is homomorphically encrypted by the local edge end and uploaded to the cloud server, and m is less than or equal to the total number of local edge ends; S22, the cloud server decrypts and performs weighted average calculation on the backbone network parameter change amount of the m local edge end models; S23, the cloud server updates the cloud global model according to the following formula: wherein, denotes the cloud global model of the s+1th round, denotes the local edge model of the s+1th round, n m is the data volume of the sampled local edge model in the current round, n all denotes the total data volume of all local edge models. 3.The personnel flow density statistics and alarm processing method based on federated learning according to claim 1, wherein, The backbone network of the local edge model is uploaded to the cloud server, and the classifier of the local edge model is only updated locally; wherein, denotes the local edge model of the s-th round, denotes the classifier of the local edge model of the s-th round, and η denotes the learning rate, denotes the classifier of the local edge model of the s-th round, and η denotes the learning rate, the computed gradient. 4.The personnel flow density statistics and alarm processing method based on federated learning according to claim 1, wherein, The step S531 further includes the following steps: inputting the to-be-tested monitoring image into a DLA-34 network of the local edge model for feature extraction, and outputting four branch feature maps, wherein the size of the input to-be-tested monitoring image is represented as H image ×W image , and the shape of the output feature map is H×W×C, and (H,W) is the pixel of the to-be-tested monitoring image, and C is the channel number of the to-be-tested monitoring image. 5.The personnel flow density statistics and alarm processing method based on federated learning according to claim 1, wherein, The step S533 further includes the following steps: Residual network is used for feature extraction of the feature map to obtain a feature vector; The feature vector is converted into one-hot encoding which can construct a loss function through one or more fully connected layers; A loss function is constructed using the cross-entropy method; The cosine distance is used to calculate the distance between the feature vectors to evaluate the performance of the local model; The K-Nearest Neighbor method is used to select the top K most matched images to obtain a ranking list to evaluate the accuracy of the local model. 6.The personnel flow density statistics and alarm processing method based on federated learning according to claim 1, wherein, The step S60 further includes the following steps: S61, obtaining the personnel flow density statistical result; S62, judging whether the personnel flow density reaches the alarm threshold, and generating an alarm work order when the personnel flow density reaches the alarm threshold.

7. A personnel flow density statistics and alarm processing system based on federated learning, characterized in that, The personnel flow density statistical and alarm processing system based on federated learning includes a cloud server and multiple local edge ends, wherein the cloud server includes a global model training module and a training termination judgment module, and each local edge end includes a local model training module, a people flow density statistical module, and an alarm module; The global model training module is configured to aggregate the backbone network parameter change amount of the local edge model from the local edge end, and update the cloud global model using the aggregated backbone network parameter change amount of the local edge model; The local model training module is configured to train the FairMOT network using the monitoring video data to obtain the local edge model, and download the parameter change amount of the cloud global model from the cloud server, combine the classifier of the local edge model, and use the monitoring video data to perform a new round of local training; The training termination judgment module is configured to judge whether the cloud global model meets the convergence requirement, and judge whether to terminate the training of the cloud global model and the local edge model according to the judgment result; The people flow density statistical module is configured to identify people and count the personnel flow density using the trained local edge model on real-time monitoring video data, and output the personnel flow density statistical result; The alarm module is configured to match the personnel flow density statistical result with a preset people flow density alarm strategy, and generate an alarm work order when the personnel flow density reaches an alarm trigger point; The people flow density statistical module is specifically configured to: Obtain real-time monitoring video data and perform preprocessing to generate real-time monitoring images that meet the model input requirements; Identify the time of the real-time monitoring image using an OCR algorithm, determine the real-time monitoring image within the statistical range of the current rolling time window according to the time of the real-time monitoring image, and record it as a to-be-tested monitoring image; Identify people and count the personnel flow density using the trained local edge model on the to-be-tested monitoring image, and output the personnel flow density statistical result; When the current rolling time window ends, end the current statistical round, jump to step S51, and start a new statistical round; The trained local edge model is used for personnel recognition and personnel flow density statistics on the to-be-tested monitoring image, and outputs a personnel flow density statistical result, specifically including: The to-be-tested monitoring image is input into a backbone network of the local edge model for feature extraction, and four branch feature maps are output. The feature maps of the three branches are input into three parallel heads of a multi-parallel head module of a target detection network of the local edge model, and the three parallel heads respectively output the center position, the recognition frame size and the center offset of each detection object. The feature map of the one branch is input into a Re-ID image reclassification branch structure in the local edge model, and object Re-ID feature extraction is performed for personnel re-identification tracking.

Images