Federated Learning Method, Identification Method and Device for Public Safety Information Identification Model

Abstract

The present application provides a federated learning method, an identification method and a device for a public safety information identification model. The method includes: receiving the global model of the previous round and the respective fine-grained global prototypes of each fine-grained class under each class, and performing hierarchical prototype contrast learning, batch prototype regularization and global information distillation processing based on the local monitoring image samples, public safety information identification labels, the public safety information identification global model of the previous round, the public safety information identification local model and each fine-grained local prototype, and performing gradient optimization on the local public safety information identification local model of the current round based on the complete loss function. The present application can solve the class imbalance problem in Non-IID federated learning, more accurately rebalance the feature distribution of samples on the client, effectively reduce the bias of the local classifier, and enhance the robustness of the aggregated global classifier, thereby effectively improving the accuracy and effectiveness of public safety information identification.

Description

Technical Field

[0001] This application relates to the field of public safety technologies, and in particular, to a federated learning method, an identification method, and a device for a public safety information identification model. Background Art

[0002] In order to more conveniently maintain public safety, we need to more accurately identify the risky content of public safety information from various channels. With the rapid development of computer vision, machine learning has been closely linked to the automated identification of public safety information. However, considering that the cameras in different regions may belong to the assets of different enterprises and other entities, it is not convenient to share the monitoring data with other entities. Therefore, the application of federated learning in the identification of public safety information has emerged, and it can also be called visual federation. This method can not only avoid sharing its own public safety monitoring data with other entities, but also use the public safety monitoring data of other entities to improve the accuracy of the public safety information identification model it will use. However, since the data mastered by different entities usually has the problem of data class imbalance, the traditional federated learning algorithm tends to the class with a larger number of samples during the client update, resulting in client drift. In addition, existing research shows that the classification layer in the client deep model introduces more biases than the hidden layer in non-independent and identically distributed federated learning, which significantly deviates from the global optimum in terms of the client optimization objective.

[0003] Currently, many federated learning methods have been proposed by researchers to solve the problem of non-independent and identically distributed federated learning. Classic methods include Fedprox, Scaffold, FedDC, Lumos, and FedIR, which propose local optimization constraints to coordinate local and global optimization objectives. Other methods include FedNova, FedMA, and CCVR, which make the global model closer to the global optimum by improving the global aggregation stage. Although these methods have made progress, they still have deficiencies in addressing the challenge of classifier bias brought about by imbalanced classes.

[0004] First, the main reason for this problem is that the client cannot obtain the distribution information of the global sample features during local update. Therefore, the learned feature representation may lack highly separable features. Second, during the local update process, the gradual loss of global classification information leads to the local classifier tending to the local optimum. Summary of the Invention

[0005] In view of this, the embodiments of this application provide a federated learning method, an identification method, and a device for a public safety information identification model to eliminate or improve one or more defects existing in the prior art.

[0006] One aspect of the present application provides a federated learning method for a public safety information recognition model, including:

[0007] Receiving the global model of public safety information recognition in the previous round sent by the server, and the fine-grained global prototypes of each fine-grained class under each class in the previous round;

[0008] Initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performing preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public safety information recognition label, the global model of public safety information recognition in the previous round, the local model of public safety information recognition locally, and each of the fine-grained local prototypes, and performing gradient optimization on the local model of public safety information recognition locally in the current round based on a preset complete loss function to obtain the local model of public safety information recognition in the current round as the target and each target fine-grained local prototype;

[0009] Sending the target local model of public safety information recognition and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target local models of public safety information recognition received in the current round to obtain the global model of public safety information recognition in the current round.

[0010] In some embodiments of the present application, the global model of public safety information recognition includes a global feature extractor, a global classifier, and an aggregation matrix connected in sequence; the local model of public safety information recognition includes a local feature extractor, a local classifier, and an aggregation matrix connected in sequence;

[0011] Correspondingly, the initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performing preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public safety information recognition label, the global model of public safety information recognition in the previous round, the local model of public safety information recognition locally, and each of the fine-grained local prototypes includes:

[0012] Extracting the sample features corresponding to each local monitoring image sample based on the local feature extractor, and respectively assigning each of the monitoring image samples to each fine-grained class locally according to the sample features;

[0013] Initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round;

[0014] Using the sample features corresponding to each local monitoring image sample to perform hierarchical prototype contrast learning on the fine-grained local prototypes of the current local round to calculate the hierarchical prototype contrast learning loss;

[0015] Obtaining the corresponding batch prototypes according to the sample features corresponding to each local monitoring image sample, and performing regularization processing on the batch prototypes according to the preset public security information recognition label, the global classifier, and the aggregation matrix to calculate the batch prototype regularization loss;

[0016] Based on the public security information recognition global model and the public security information recognition local model of the previous round, respectively obtaining the prediction probabilities corresponding to each local monitoring image sample, and calculating the global information distillation loss based on the prediction probabilities respectively output by the public security information recognition global model and the public security information recognition local model;

[0017] And calculating the debiased classifier learning loss according to the prediction probability output by the public security information recognition local model and the preset public security information recognition label.

[0018] In some embodiments of the present application, the gradient optimization of the local public security information recognition local model of the current round based on the preset complete loss function to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype includes:

[0019] Determining the value of the anti-overfitting parameter of the current round based on the number of the current round in the total number of iteration rounds and the number of the total iteration rounds;

[0020] Solving the complete loss function according to the value of the anti-overfitting parameter of the current round, the preset hyperparameters, the hierarchical prototype contrast learning loss, the batch prototype regularization loss, the global information distillation loss, and the debiased classifier learning loss, and performing gradient optimization on the local public security information recognition local model of the current round based on this complete loss function, so as to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype.

[0021] In some embodiments of the present application, the using the sample features corresponding to each local monitoring image sample to perform hierarchical prototype contrast learning on the fine-grained local prototypes of the current local round to calculate the hierarchical prototype contrast learning loss includes:

[0022] Calculating the hierarchical prototype contrast learning loss L based on the following formula (1) hpc :

[0023]

[0024] Among them,

[0025]

[0026]

[0027]

[0028]

[0029] In the above formulas (1) to (5), N is the sum of important factors; is the sample feature; D k is the local data set composed of local monitoring image samples; I c,t is the important factor; is the sample feature optimization parameter; Ψ pos is the positive pair corresponding to the monitoring image sample; α is the feature hierarchy distance adjustment parameter; Ψ pneg is the pseudo-negative pair corresponding to the monitoring image sample; Ψ neg is the negative pair corresponding to the monitoring image sample; is the fine-grained local prototype corresponding to the t-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the j-th class.

[0030] In some embodiments of the present application, the process of the server aggregating all the local models for identifying target public security information received in the current round to obtain the global model for identifying public security information in the current round includes:

[0031] The server aggregates the local feature extractors in all the local models for identifying target public security information received in the current round to obtain the global feature extractor in the current round;

[0032] And, the server performs adaptive class-level classifier aggregation on the local classifiers in all the local models for identifying target public security information received in the current round to obtain the global classifier in the current round;

[0033] The server generates the global model for identifying public security information in the current round based on the global feature extractor, global classifier, and the aggregation matrix in the current round.

[0034] Another aspect of the present application provides a method for identifying public security information, including:

[0035] Obtain the monitoring image data of the target area;

[0036] Input the monitoring image data into the global public security information recognition model obtained by the federated learning method of the public security information recognition model based on the above, so as to determine the public security information recognition result of the monitoring image data according to the output of the global public security information recognition model.

[0037] The third aspect of this application provides a federated learning device for a public security information recognition model, including:

[0038] A global data receiving module, configured to receive the global public security information recognition model of the previous round sent by the server, and the fine-grained global prototypes of each fine-grained class under each class in the previous round.

[0039] A model training module, configured to initialize the fine-grained local prototypes of the current local round based on the fine-grained global prototypes of the previous round, and perform preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to local monitoring image samples, preset public security information recognition labels, the global public security information recognition model of the previous round, the local public security information recognition local model, and each of the fine-grained local prototypes, and perform gradient optimization on the local public security information recognition local model of the current round based on a preset complete loss function to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype.

[0040] A local data sending module, configured to send the target public security information recognition local model and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target public security information recognition local models received in the current round to obtain the global public security information recognition model of the current round.

[0041] The fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the federated learning method of the public security information recognition model, and / or implements the public security information recognition method.

[0042] The fifth aspect of this application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the federated learning method of the public security information recognition model, and / or implements the public security information recognition method.

[0043] The sixth aspect of the present application provides a federated learning system, including: a server, and each client device communicatively connected to the server;

[0044] The client device is used to execute the federated learning method of the public security information recognition model, and / or, is used to execute the public security information recognition method;

[0045] The server includes:

[0046] A local data receiving module, configured to receive the target public security information recognition local model and each target fine-grained local prototype of the current round respectively sent by each of the client devices;

[0047] A fine-grained global prototype aggregation module, configured to aggregate all the target fine-grained local prototypes received in the current round to obtain the respective fine-grained global prototypes of each fine-grained class under each class in the current round;

[0048] A feature extractor aggregation module, configured to aggregate the local feature extractors in all the target public security information recognition local models received in the current round to obtain the global feature extractor of the current round;

[0049] An adaptive class-level classifier aggregation module, configured to perform adaptive class-level classifier aggregation on the local classifiers in all the target public security information recognition local models received in the current round to obtain the global classifier of the current round;

[0050] A global data sending module, configured to generate the global model for public security information recognition of the current round based on the global feature extractor, global classifier and aggregation matrix of the current round. If the current round is not the last round in the total number of iterative rounds, the global model for public security information recognition of the current round and the respective fine-grained global prototypes of each fine-grained class under each class in the current round are respectively sent to each of the client devices.

[0051] The federated learning method for the public security information recognition model provided by this application receives the global model of public security information recognition in the previous round sent by the server and the fine-grained global prototypes of each fine-grained class under each class in the previous round; initializes the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performs preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the global model of public security information recognition in the previous round, the local model of public security information recognition locally, and each of the fine-grained local prototypes, and optimizes the gradient of the local model of public security information recognition in the current round based on the preset complete loss function to obtain the target local model of public security information recognition in the current round and each target fine-grained local prototype; sends the target local model of public security information recognition and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target local models of public security information recognition received in the current round to obtain the global model of public security information recognition in the current round. Hierarchical prototype contrast learning and batch prototype regularization are proposed to learn fine-grained prototypes, which can accurately represent the global sample distribution and promote the aggregation of feature representations within classes, while maintaining a distance from other classes. Compared with the single prototype method, the proposed fine-grained prototypes more accurately rebalance the feature distribution of samples on the client, effectively reducing the bias of the local classifier; through global information distillation, the local and global classifiers are adjusted at the decision level, thereby reducing the bias of the local classifier towards a large number of classes on the client. By decoupling the soft labels output by the global classifier, the local classifier can eliminate bias through global classification information that is not accessible to the client, and thus can solve the class imbalance problem in Non-IID federated learning, more accurately rebalance the feature distribution of samples on the client, effectively reduce the bias of the local classifier, and enhance the robustness of the aggregated global classifier, thereby effectively improving the accuracy and effectiveness of public security information recognition.

[0052] Additional advantages, objects, and features of this application will be partly described in the following description, and will partly become apparent to those of ordinary skill in the art after studying the following part, or can be learned from the practice of this application. The objects and other advantages of this application can be achieved and obtained by the structure specifically pointed out in the specification and the drawings.

[0053] Those skilled in the art will understand that the objects and advantages that can be achieved by this application are not limited to the above specifically described, and the above and other objects that can be achieved by this application will be more clearly understood according to the following detailed description. Brief Description of the Drawings

[0054] The drawings described herein are provided to further understand the present application, form a part of the present application, and do not limit the present application. The components in the drawings are not drawn to scale, but are only for showing the principles of the present application. To facilitate showing and describing some parts of the present application, corresponding parts in the drawings may be enlarged, that is, may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0055] Figure 1 It is a first process schematic diagram of the federated learning method for the public safety information recognition model in an embodiment of the present application.

[0056] Figure 2 It is a second process schematic diagram of the federated learning method for the public safety information recognition model in an embodiment of the present application.

[0057] Figure 3 It is a schematic diagram of a new federated learning framework introduced in an application example of the present application.

[0058] Figure 4 It is a process schematic diagram of the public safety information recognition method in an embodiment of the present application.

[0059] Figure 5 It is a structural schematic diagram of the federated learning device for the public safety information recognition model in an embodiment of the present application.

[0060] Figure 6 It is a schematic diagram of the communication connection relationship between the federated learning device for the public safety information recognition model and the server in an embodiment of the present application. Detailed Description of the Embodiments

[0061] To make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the embodiments and the drawings. Herein, the illustrative embodiments and descriptions thereof of the present application are used to explain the present application, but do not limit the present application.

[0062] Herein, it should also be noted that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the solution according to the present application are shown in the drawings, while other details less related to the present application are omitted.

[0063] It should be emphasized that the term "including / comprising" when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0064] Here, it should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to direct connection but also to indirect connection with intermediaries.

[0065] In the following, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0066] Traditional federated learning methods, such as FedAvg, have achieved remarkable success in scenarios where data is independently and identically distributed (IID). However, in real-world scenarios, data samples from different clients usually exhibit non-IID characteristics. One of the key challenges in non-IID federated learning is the existence of class imbalance. The existence of data imbalance may cause FedAvg to bias towards classes with a larger number of samples during client updates, resulting in client drift. In addition, existing research has shown that the classification layer in the client deep model introduces more bias than the hidden layer in non-IID federated learning, which significantly deviates from the global optimum in terms of the client optimization objective. Therefore, in this case, the performance of FedAvg will deteriorate significantly. This problem is also an urgent problem to be solved in non-IID federated learning.

[0067] Researchers have proposed many federated learning methods to solve the non-IID federated learning problem: 1) Increasing local optimization constraints to coordinate local and global optimization objectives: Some methods attempt to alleviate the degree of local client deviation by designing local loss functions during local updates. 2) Improving the global aggregation strategy to make the global model closer to the global optimum: Some methods make the global model closer to the global optimum by post-calibrating the global model using server resources. Some methods design better aggregation strategies to improve the performance of the aggregated global model by matching and averaging weights and constructing a shared global model in a hierarchical manner.

[0068] Some federated learning algorithms mitigate client drift by modifying the local optimization objective. For example, Fedprox directly uses the l2-norm distance to mitigate client drift, Scaffold uses variance reduction to correct client drift, FedDC uses learned local drift variables to bridge the gap, i.e., performing consistency constraints at the parameter level. FedDyn dynamically iteratively optimizes the local objective to achieve asymptotic consistency between the local optimum and the global objective stationary point, and FedIR applies importance weights to the local objective to mitigate the imbalance caused by different class distributions among clients. In addition, enhancing the model aggregation phase is also used to mitigate the adverse impact of client drift on the global model. FedNova introduces a regularization term based on local steps to limit the impact on the global model, FedMA matches and averages weights to construct a shared model for the global model in a hierarchical manner, FedAvgM uses momentum optimization on the server to improve robustness against different client data distributions, and CCVR samples pseudo-samples on the server and calibrates the classifier to address the classifier drift problem.

[0069] In recent years, prototype learning has been widely developed, where class prototypes are represented as the average feature vectors of classes. FedProto proposes minimizing the communication overhead by exchanging prototypes instead of gradients or model parameters between clients and the server. FedProc introduces the utilization of global prototypes on the server as a reference for refining client training during local updates. It adopts contrastive loss to encourage features within a class to be closer while features between classes to be farther apart. MP-FedCL combines multiple prototypes with contrastive learning and uses prototypes instead of classifiers for prediction.

[0070] Researchers have proposed many federated learning methods to address the non-i.i.d. federated learning problem. Classic methods include Fedprox, Scaffold, FedDC, Lumos, and FedIR, which propose local optimization constraints to coordinate local and global optimization objectives. Other methods include FedNova, FedMA, and CCVR, which make the global model approach the global optimum by improving the global aggregation phase. Although these methods have made progress, they still have deficiencies in addressing the challenge of classifier bias caused by imbalanced classes. In the following analysis, the limitations of these methods will be discussed to clarify the root cause of this problem.

[0071] First, the main reason for this problem is that clients cannot obtain the distribution information of global sample features during local updates. Therefore, the learned feature representations may lack highly separable features. Second, during the local update process, the gradual loss of global classification information leads to the local classifier being biased towards the local optimum.

[0072] Based on this, in order to improve the application accuracy and effectiveness of the public safety information recognition model, different from previous methods, this application uses fine-grained prototypes optimized by hierarchical prototype contrast learning to solve the problem of unbalanced sample distribution on the client side. Knowledge distillation is performed on the client side to instruct the local classifier to obtain more global classification information. In the global aggregation stage, this application introduces class-level aggregation of the classifier instead of the overall classifier aggregation method.

[0073] Specifically, it is described in detail through the following embodiments.

[0074] Based on this, an embodiment of this application provides a federated learning method for a public safety information recognition model that can be implemented by a federated learning device of the public safety information recognition model. Refer to Figure 1 , the federated learning method for the public safety information recognition model specifically includes the following content:

[0075] Step 100: Receive the global model of public safety information recognition in the previous round sent by the server and the fine-grained global prototypes of each fine-grained class under each class in the previous round.

[0076] It can be understood that the functions of the federated learning device of the public safety information recognition model can be specifically implemented in a client device, and this client device and multiple other client devices are all in the same federated learning system as the server.

[0077] In one or more embodiments of this application, the model architectures of the global model of public safety information recognition, the local model of public safety information recognition, and the target local model of public safety information recognition are exactly the same, and they are all the public safety information recognition models mentioned in this application. These different name expressions are only used to distinguish different states of the public safety information recognition model.

[0078] Step 200: Initialize the fine-grained local prototypes of the current local round based on the fine-grained global prototypes of the previous round, and perform preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public safety information recognition label, the global model of public safety information recognition in the previous round, the local model of public safety information recognition of the local, and each of the fine-grained local prototypes, and perform gradient optimization on the local model of public safety information recognition of the current round based on the preset complete loss function to obtain the target local model of public safety information recognition of the current round and each target fine-grained local prototype.

[0079] It is understandable that this application aims to solve the class imbalance problem in Non-IID federated learning. The existence of class imbalance often leads to insufficient samples of minority classes. This imbalance poses challenges to accurately representing the true data distribution and also hinders the effective extraction of relevant information during the local model training process. In addition, in the case of class imbalance, local classifiers tend to show bias towards the majority classes, resulting in a decline in the overall performance of the global model. In the true data distribution, samples of the same class cannot be compactly clustered in one cluster, making the learning of a single prototype insufficient to effectively describe the sample distribution of each class. To overcome these challenges, this application proposes a new method, including learning fine-grained prototypes for each class. These fine-grained prototypes accurately capture the sample distribution of each class on the client side and are used to mitigate the bias of local classifiers.

[0080] Meanwhile, this application proposes a hierarchical prototype contrastive learning loss to learn the fine-grained prototypes of each class, enabling the fine-grained prototypes to more accurately depict the sample distribution.

[0081] It is understandable that the monitored image samples can specifically be taken from monitored camera images, Internet and social network images, etc. authorized to be obtained by relevant security management departments, institutions or units. The public security information recognition labels can be set according to actual application requirements. For example, for the public security information recognition requirements of the monitored object, the labels can be set as at least two of violent behavior, fire, bad weather, robbery, and normal, etc. These labels correspond one by one to each class. That is to say, each class or category mentioned in this application refers to the true category, such as category "1", etc.

[0082] Step 300: Send the target public security information recognition local model and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target public security information recognition local models received in the current round to obtain the public security information recognition global model in the current round.

[0083] In a scenario after step 300 is executed, if the current round is already the last round of the total number of iterations, the federated learning process ends, and the server distributes the globally recognized model of public security information for the current round obtained finally to each of the client devices as a globally recognized model of public security information that can be used for online applications; if the current round is not the last round of the total number of iterations, then the server distributes the globally recognized model of public security information for the current round and the fine-grained global prototypes of each fine-grained class under each class in the current round to each client device, so that each client device returns to execute step 100.

[0084] In addition, if the current round is not the last round of the total number of iterations, but the client device receives an online application request for the globally recognized model of public security information during any step from step 100 to step 300, it can first use the globally recognized model of public security information for the previous round received in step 100 or the globally recognized model of public security information generated in the current round sent by the server received after step 300 as a globally recognized model of public security information that can be used for online applications, and perform the recognition of public security information for the monitoring image data of the target area based on this globally recognized model of public security information.

[0085] As can be seen from the above description, the federated learning method for the globally recognized model of public security information provided in the embodiments of the present application proposes hierarchical prototype contrast learning and batch prototype regularization to learn fine-grained prototypes, and these prototypes can accurately represent the global sample distribution, promote the aggregation of feature representations within a class, and at the same time keep a distance from other classes. Compared with the single prototype method, the proposed fine-grained prototypes can more accurately rebalance the feature distribution of samples on the client, effectively reducing the bias of the local classifier; through global information distillation, the local and global classifiers are adjusted at the decision level, thereby reducing the bias of the local classifier for a large number of classes on the client. By decoupling the soft labels output by the global classifier, the local classifier can eliminate the bias through the global classification information inaccessible to the client, and thus can solve the class imbalance problem in non-IID federated learning, more accurately rebalance the feature distribution of samples on the client, effectively reduce the bias of the local classifier, and enhance the robustness of the aggregated global classifier, and thus can effectively improve the accuracy and effectiveness of public security information recognition.

[0086] In order to further improve the effectiveness and applicability of hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing, in a federated learning method for a globally recognized model of public security information provided in the embodiments of the present application, the globally recognized model of public security information includes a global feature extractor, a global classifier, and an aggregation matrix connected in sequence; the locally recognized model of public security information includes a local feature extractor, a local classifier, and an aggregation matrix connected in sequence;

[0087] Correspondingly, refer to Figure 2 , step 200 in the federated learning method of the public security information recognition model specifically includes the following content:

[0088] Step 210: Extract sample features corresponding to each local monitoring image sample based on the local feature extractor, and assign each of the monitoring image samples to each fine-grained class locally according to the sample features;

[0089] Specifically, after receiving the fine-grained global prototype from the server, continue to assign the samples to the fine-grained classes on the client.

[0090] Step 220: Initialize the fine-grained local prototype of the current local round according to the fine-grained global prototype of the previous round;

[0091] Step 230: Perform hierarchical prototype contrast learning on the fine-grained local prototype of the current local round using the sample features corresponding to each local monitoring image sample to calculate the hierarchical prototype contrast learning loss;

[0092] Step 240: Obtain the corresponding batch prototypes according to the sample features corresponding to each local monitoring image sample, and perform regularization processing on the batch prototypes according to the preset public security information recognition label, the global classifier, and the aggregation matrix to calculate the batch prototype regularization loss;

[0093] Step 250: Obtain the prediction probabilities corresponding to each local monitoring image sample locally based on the public security information recognition global model and the public security information recognition local model of the previous round, and calculate the global information distillation loss based on the prediction probabilities respectively output by the public security information recognition global model and the public security information recognition local model;

[0094] And, step 260: Calculate the debiased classifier learning loss according to the prediction probability output by the public security information recognition local model and the preset public security information recognition label.

[0095] In order to further improve the robustness and reliability of the federated learning of the public security information recognition model, in a federated learning method of a public security information recognition model provided in an embodiment of the present application, refer to Figure 2 , step 200 in the federated learning method of the public security information recognition model further specifically includes the following content:

[0096] Step 270: Determine the value of the overfitting prevention parameter of the current round based on the number of the current round in the total number of iteration rounds and the number of the total iteration rounds.

[0097] Step 280: Solve the complete loss function according to the value of the anti-overfitting parameter in the current round, the preset hyperparameters, the hierarchical prototype contrastive learning loss, the batch prototype regularization loss, the global information distillation loss, and the debiased classifier learning loss, so as to optimize the gradient of the local public safety information recognition local model in the current round based on this complete loss function, and then obtain the target public safety information recognition local model in the current round and each target fine-grained local prototype.

[0098] To further enhance the hierarchical separability between fine-grained categories, in a federated learning method for a public safety information recognition model provided in an embodiment of the present application, step 230 in the federated learning method for the public safety information recognition model specifically includes the following contents:

[0099] Calculate the hierarchical prototype contrastive learning loss L based on the following formula (1) hpc :

[0100]

[0101] Wherein,

[0102]

[0103]

[0104]

[0105]

[0106] In the above formulas (1) to (5), N is the sum of important factors; is the sample feature; D k is the local dataset composed of local monitoring image samples; I c,t is the important factor; is the sample feature optimization parameter; Ψ pos is the positive pair corresponding to the monitoring image sample; α is the feature hierarchical distance adjustment parameter; Ψ pneg is the pseudo-negative pair corresponding to the monitoring image sample; Ψ neg is the negative pair corresponding to the monitoring image sample; is the fine-grained local prototype corresponding to the t-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the j-th class.

[0107] That is to say, compared with the traditional contrastive learning loss, the hierarchical contrastive learning loss adopted in this application explicitly distinguishes true negative pairs and pseudo-negative pairs, enhancing the hierarchical separability between fine-grained categories.

[0108] In Figure 3 P k,c×m represents the probability that the k-th client classifies the prototype into the c×m-th fine-grained category, where k represents the client number and c×m represents the fine-grained category number.

[0109] In addition, regarding the technical problem still existing in the prior art: in the global aggregation stage, the different importance of the knowledge learned by the same client for different categories is not considered, which may further cause low-quality categories in the client to have a negative impact on the aggregation process of the global model. To solve this technical problem, in a federated learning method of a public security information recognition model provided in an embodiment of this application, the process by which the server aggregates all the received target local public security information recognition models in the current round to obtain the global public security information recognition model in the current round includes:

[0110] The server aggregates the local feature extractors in all the received target local public security information recognition models in the current round to obtain the global feature extractor in the current round;

[0111] And, the server performs adaptive class-level classifier aggregation on the local classifiers in all the received target local public security information recognition models in the current round to obtain the global classifier in the current round;

[0112] The server generates the global public security information recognition model in the current round based on the global feature extractor, global classifier, and the aggregation matrix in the current round.

[0113] To further illustrate the complete process of the above-mentioned federated learning method of the public security information recognition model, this application also provides a specific application example of the federated learning method of the public security information recognition model. Refer to Figure 3 , this application example introduces a new federated learning framework to address classifier bias in non-independent and identically distributed federated learning. This method integrates fine-grained prototype learning, global information knowledge distillation, and adaptive class-level classifier aggregation within a unified framework. By correcting classifier bias at different stages of federated learning, including representation and classifier learning on the client side, and global aggregation on the server side, this application example outperforms existing Non-IID federated learning methods, and the specific advantages are as follows:

[0114] 1. The application example of this application introduces hierarchical prototype contrast learning, enabling each class to obtain fine-grained prototypes, which can better capture the global distribution compared to using a single prototype.

[0115] 2. The application example of this application uses global information distillation to decouple global prediction information, effectively correcting local classifier bias.

[0116] 3. The application example of this application evaluates the quality of class vectors in the classifier, thereby reducing the impact of classifier bias on model aggregation accuracy during the server aggregation stage.

[0117] In the application example of this application, the local model for public safety information recognition can be abbreviated as the local model, the global model for public safety information recognition can be abbreviated as the global model, the fine-grained global prototype can be abbreviated as the global prototype, and the fine-grained local prototype can be abbreviated as the local prototype. It should be noted that in this application, "local" refers to the local of the client device. When the local model or local prototype generated by the client device is sent to the server, it still refers to the model or prototype generated locally by the client device, rather than the local of the server.

[0118] The application example of this application provides a federated learning method for a public safety information recognition model, which specifically includes the following content:

[0119] I. Aggregation of Fine-Grained Global Prototypes

[0120] In this task, all samples belong to C true classes. Each class is further divided into M fine-grained classes, and each fine-grained class contains a fine-grained prototype P c,m , where c ∈ [1, C], m ∈ [1, M]. In the global aggregation stage, the goal of the application example of this application is to represent the global data distribution information by aggregating the fine-grained global prototypes P g of each class. To achieve this goal, a weighted aggregation method is adopted on the server to aggregate the fine-grained local prototypes from each client. The weight assigned for aggregation is determined by the ratio of the number of samples in the class to the total number of samples in the class across all clients. Each fine-grained local prototype is obtained through gradient descent optimization in the local update stage.

[0121] When the server receives all the local fine-grained prototypes belonging to the m-th fine-grained class under the c-th true class, the fine-grained global prototype corresponding to the m-th fine-grained class under the c-th class is obtained

[0122]

[0123] where is the fine-grained local prototype from the k-th client, that is, the fine-grained local prototype and n c,m respectively represent the number of samples of the corresponding fine-grained class on the k-th client and the total number of samples of this class on all clients.

[0124] II. Fine-grained prototype learning

[0125] The application example of this application aims to solve the class imbalance problem in Non-IID federated learning. The existence of class imbalance often leads to insufficient samples of minority classes. This imbalance poses challenges to accurately representing the true data distribution and also hinders the effective extraction of relevant information in the local model training process. In addition, in the case of class imbalance, local classifiers tend to show bias towards the majority classes, resulting in a decline in the overall performance of the global model. In the true data distribution, samples of the same class cannot be compactly clustered in one cluster, resulting in the learning of a single prototype being insufficient to effectively describe the sample distribution of each class. To overcome these challenges, the application example of this application proposes a new method, including learning fine-grained prototypes for each class. These fine-grained prototypes accurately capture the sample distribution of each class on the client and are used to mitigate the bias of local classifiers. Specifically, it includes the following content:

[0126] 2.1. Fine-grained class assignment

[0127] After receiving the fine-grained global prototype P g from the server, continue to assign samples to the fine-grained classes on the client. For the i-th sample in the c-th class on the k-th client whose feature representation is obtain its fine-grained class as follows:

[0128]

[0129]

[0130]

[0131] where sim represents the similarity function, t represents the t-th fine-grained class. θ k represents the parameters of the feature extractor. represents the sample on the k-th client and the similarity with the m-th fine-grained class. represents the sample feature and the set of similarities with all the fine-grained prototypes of the c-th class. For where k represents the k-th client, m represents the m-th fine-grained class in the current c-th class.

[0132] 2.2. Hierarchical prototype contrastive learning

[0133] Compared with the existing method that uses the average of class samples as the prototype, the application example of this application uses fine-grained global prototypes to initialize fine-grained local prototypes (in the first round, k-means is used to initialize local prototypes), and regards them as learnable variables, which are optimized by gradient descent. Specifically, given a sample feature it should be closest to the fine-grained local prototype it belongs to while maintaining an appropriate distance from other fine-grained local prototypes in the same class and showing a significant difference from the local prototypes of different classes P j,m , j≠c. To achieve this, a new hierarchical prototype contrast learning method is proposed, constructing a positive pair Ψ pos , M-1 pseudo-negative pairs Ψ pneg and (C-1)×M negative pairs Ψ neg to align sample representations and promote the learning of accurate fine-grained prototypes to capture the sample distribution. The loss is defined as follows:

[0134]

[0135]

[0136]

[0137] where τ is the temperature parameter. A feature hierarchical distance adjustment parameter α is introduced to adjust the hierarchical distance of features to two types of negative pairs. The loss of hierarchical prototype contrast learning is defined as follows:

[0138]

[0139] where,

[0140]

[0141] where is the rebalanced local dataset. D k is the local sample feature set, P g is the fine-grained global prototype set. The important factor I c,t represents the importance of each sample. For I c,t , the local sample is assigned 1. For the fine-grained global prototype I c,t is equal to n c,t . is the sum of the important factors. By setting α < 0.5, it means that the feature is farther from the true negative sample compared to the pseudo-negative sample. is the sample feature optimization parameter.

[0142] 2.3. Batch Prototype Regularization

[0143] Enhancing the separability of feature representations is crucial for classifier calibration. In view of this, the application example of this application introduces batch prototype regularization to effectively align features with the input space of the global classifier. By incorporating this regularization loss, the separability of feature representations can be enhanced, thereby improving the overall discriminative ability:

[0144]

[0145] where D is the global classifier and φ are its parameters. W ∈ R (C×M)×C is an aggregation matrix used to integrate the classification results output by the fine-grained classifier. σ is the softmax function. is the batch prototype (batch feature mean) for class c, and N b is the number in the batch . Y b is the label set of this batch of samples. d i represents the probability distribution output by the global classifier for the batch prototype. Note that the sets of and Y b for different batches are different. The batch prototype regularization loss formula is:

[0146]

[0147] where, is the c-th element of d i , and c represents the true class of the i-th batch of prototypes.

[0148] III. Global Information Distillation

[0149] During local updates, the goal of the application example of this application is to transfer the knowledge of the global model to the local model using the global model to ensure consistency between the local model and the global model. The classifier output contains two parts: target class information and non-target class information. Traditional knowledge distillation methods often ignore the importance of non-target class information, which is the key to effective distillation. In a federated scenario where client data access is restricted, it is particularly important to recognize the importance of non-target class information. The global classification information in non-target classes is the key to reducing classifier bias. Decoupled Knowledge Distillation (DKD) divides traditional knowledge distillation into Target Class Knowledge Distillation (TCKD) and Non-Target Class Knowledge Distillation (NCKD). Inspired by this, decoupled knowledge distillation is adopted to mitigate the occurrence of classifier drift caused by class imbalance. This is achieved by enabling the local model to obtain a richer understanding of global information through NCKD. The DKD loss formula is:

[0150]

[0151] After receiving a sample p T represents the probability distribution output by the global model, p S represents the probability distribution output by the local model. where c represents the true category of the sample, that is represents the probability that the global model assigns the sample to category c, b S is the same as b T Similarly. represents the result of normalizing the probability distribution output by the global model after removing . Similarly.

[0152] The global information distillation loss is defined as follows:

[0153]

[0154] where and represent the predicted probability outputs of the global model and the local model, respectively.

[0155] IV. Biased classifier learning

[0156] To solve the classifier drift problem on class-imbalanced clients, the application instance of this application proposes a debiased classifier learning loss L dcl , including using fine-grained global prototypes to rebalance the distribution of client samples. Through these prototypes, the global distribution information can be effectively utilized to reduce the bias in the classifier, and the loss function is defined as follows:

[0157]

[0158] Finally, by integrating the above objectives, the complete loss function for client optimization in the FedCD (i.e., A Classifier Debiased Federated Learning Framework for Non-IID Data) method proposed in this application can be obtained:

[0159] L total = μL dcl +(1 - μ)L hpc + γL bpr + λL gid (15)

[0160] where the anti-overfitting parameter Let \(t\) be the current communication round and \(T\) be the total number of communication rounds. In the early training stage, when the feature extraction ability of the model is limited, \(\mu\) can be used to prevent the prototype from being affected by poor features. In the later stage of training, \(\mu\) is used to prevent the model from overfitting. \(\gamma\) and \(\lambda\) are hyperparameters for adjusting different loss weights.

[0161] V. Adaptive Class-Level Classifier Aggregation

[0162] In non-IID federated learning, class imbalance on clients leads to performance variations across different classes. Therefore, in the aggregation phase of the global classifier, the importance of different classifiers on individual clients is different. Adaptive class-level classifier aggregation is proposed to mitigate the adverse effects of low-quality classes on the global classifier. The application example of this application effectively addresses classifier bias by identifying the importance of different classes in clients instead of assigning a uniform weight to each client. The classifier parameters can be expressed as:

[0163]

[0164] where represents the \(j\)-th class vector after dividing the classifier parameters into \(C\times M\) class vectors,

[0165] and \(\varphi\) represents the parameters of the classifier.

[0166] The classification results (output by the client classifier) using the fine-grained global prototype are used to evaluate the importance of a specific class in a client. The following expression illustrates the classification results of the fine-grained global prototype for the \(j\)-th class, and the classifier \(D\) k is generated from the \(k\)-th client:

[0167]

[0168] where \(\varphi\) k represents the parameters of the \(k\)-th client classifier, represents the \(j\)-th fine-grained global prototype, \(j\in\)

[0169] [1, C\times M], and represents the classification result output by the \(k\)-th client classifier for the \(j\)-th fine-grained global prototype.

[0170] Then, the weights of the \(j\)-th class vector for all clients are obtained as follows:

[0171]

[0172] where represents the \(j\)-th dimension of, that is, the probability that the \(k\)-th client classifies the \(j\)-th fine-grained global prototype correctly. \(\nu\) jIt represents the set obtained by normalizing the set of probabilities that all clients correctly classify the j-th fine-grained global prototype.

[0173] Finally, to obtain an unbiased and reliable global classifier, adaptive class-level global classifier aggregation is performed based on the weights of each client's class vector as follows:

[0174]

[0175] where represents the j-th class vector in the k-th client classifier.

[0176] In specific practice, it is found that the application example of this application can effectively alleviate the classifier bias caused by class imbalance, and its performance is better than other methods. Specifically, the application example of this application has at least improved the classification accuracy by 3.17% compared with other comparison methods on the CIFAR100 dataset. The application example of this application enhances client representation learning by combining prototype learning and demonstrates better performance in various scenarios, illustrating the effectiveness of prototype learning in federated learning. Compared with the federated learning method FedProc based on traditional prototype learning, since this method only learns a single prototype for each category and cannot accurately capture the distribution information of each class, FedProc has lower performance than the application example of this application on all experimental datasets, verifying the effectiveness of the application example of this application.

[0177] Among them, the new federated learning framework introduced in the application example of this application has the following functions:

[0178] 1) Fine-grained prototype learning aligns the sample representations on each client through the loss L of hierarchical prototype contrast learning hpc to optimize each fine-grained local prototype and accurately reflect the global sample distribution. It also constrains the batch prototypes to adapt to the input position of the global classifier through the batch prototype regularization loss L bpr ;

[0179] 2) Through the use of the global information distillation loss L gid , global information distillation enables the local classifier to absorb global classification information, thus establishing consistency between the local and global classifiers. In addition, it can use the unbiased classifier learning loss L dcl to adjust the biased local classifier based on the fine-grained global prototype;

[0180] 3) Fine-grained global prototype aggregation aggregates the fine-grained local prototypes of each fine-grained class with the sample proportion of the local client as the weight into a fine-grained global prototype;

[0181] 4) Adaptive class-level classifier aggregation enhances the robustness of the global classifier by adaptively evaluating the quality of client-level class vectors using fine-grained global prototypes. This is achieved by boosting high-quality class vectors and reducing the weights of low-quality class vectors during the aggregation process.

[0182] 5) Feature extractor aggregation uses the same method as FedAvg to aggregate local feature extractors from all clients to obtain a global feature extractor.

[0183] Based on this, the application example of this application proposes a hierarchical prototype contrast learning loss and a batch prototype regularization loss to learn fine-grained prototypes, which can accurately represent the global sample distribution, promote the aggregation of intra-class feature representations, and keep distances from other classes. Compared with the single prototype method, the proposed fine-grained prototypes can more accurately rebalance the feature distribution of samples on the client, effectively reducing the bias of the local classifier.

[0184] The application example of this application proposes a global information distillation loss to adjust local and global classifiers at the decision-making level, thereby reducing the bias of the local classifier towards a large number of classes on the client. By decoupling the soft labels output by the global classifier, the local classifier can eliminate bias through global classification information inaccessible to the client.

[0185] The application example of this application introduces an adaptive class-level classifier aggregation method that divides the classifier into fine-grained class vectors. By adaptively assigning weights to these vectors, the impact of unreliable class vectors on the global classifier is effectively reduced, and the robustness of the aggregated global classifier is further enhanced.

[0186] Based on the foregoing embodiments of the federated learning method for the public security information recognition model, see Figure 4 , this application also provides an embodiment of a public security information recognition method, specifically including the following content:

[0187] Step 400: Obtain the surveillance image data of the target area.

[0188] Step 500: Input the surveillance image data into the public security information recognition global model obtained in advance based on the federated learning method of the public security information recognition model to determine the public security information recognition result of the surveillance image data according to the output of the public security information recognition global model.

[0189] In step 500, the public security information recognition global model is used to output the probability values corresponding to each public security information recognition label.

[0190] At the software level, this application also provides a federated learning device for the public security information recognition model that executes all or part of the public security information recognition model in the federated learning method. See Figure 5 The federated learning device for the public security information recognition model specifically includes the following:

[0191] A global data receiving module 10, configured to receive the public security information recognition global model of the previous round sent by the server and the fine-grained global prototypes of each fine-grained class under each class in the previous round.

[0192] A model training module 20, configured to initialize the fine-grained local prototypes of the current local round based on the fine-grained global prototypes of the previous round, and perform preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the public security information recognition global model of the previous round, the local public security information recognition local model, and each of the fine-grained local prototypes, and perform gradient optimization on the local public security information recognition local model of the current round based on a preset complete loss function to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype.

[0193] A local data sending module 30, configured to send the target public security information recognition local model and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target public security information recognition local models received in the current round to obtain the public security information recognition global model of the current round.

[0194] The embodiment of the federated learning device for the public security information recognition model provided by this application can specifically be used to execute the processing flow of the embodiment of the federated learning method for the public security information recognition model in the above embodiment. Its functions will not be elaborated here, and reference can be made to the detailed description of the embodiment of the federated learning method for the public security information recognition model above.

[0195] The part of the federated learning of the public security information recognition model by the federated learning device for the public security information recognition model can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor for the specific processing of the federated learning of the public security information recognition model.

[0196] The above-mentioned client device may have a communication module (i.e., a communication unit), which can communicate with a remote server to achieve data transmission with the server. The server may include a server on the side of the task scheduling center. In other implementation scenarios, it may also include a server of an intermediate platform, such as a server of a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed device.

[0197] Any suitable network protocol can be used for communication between the above-mentioned server and the client device, including network protocols that have not been developed as of the filing date of this application. The network protocol may, for example, include TCP / IP protocol, UDP / IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also, for example, include the RPC protocol (Remote Procedure Call Protocol) and the REST protocol (Representational State Transfer) used on top of the above-mentioned protocols.

[0198] As can be seen from the above description, the federated learning device for the public security information recognition model provided in the embodiments of this application proposes hierarchical prototype contrast learning and batch prototype regularization to learn fine-grained prototypes, which can accurately represent the global sample distribution, promote the aggregation of intra-class feature representations, and at the same time maintain a distance from other classes. Compared with the single prototype method, the proposed fine-grained prototypes can more accurately rebalance the feature distribution of samples on the client, effectively reducing the bias of the local classifier; through global information distillation, the local and global classifiers are adjusted at the decision level, thereby reducing the bias of the local classifier towards a large number of classes on the client. By decoupling the soft labels output by the global classifier, the local classifier can eliminate bias through global classification information that is not accessible to the client, and thus can solve the class imbalance problem in Non-IID federated learning, more accurately rebalance the feature distribution of samples on the client, effectively reduce the bias of the local classifier, and enhance the robustness of the aggregated global classifier, thereby effectively improving the accuracy and effectiveness of public security information recognition.

[0199] The embodiments of this application also provide an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the federated learning method and / or the public security information recognition method of the public security information recognition model mentioned in the above embodiments. The processor and the memory may be connected through a bus or other means. Taking the connection through the bus as an example. The receiver can be connected to the processor and the memory in a wired or wireless manner.

[0200] The processor may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., such as chips, or combinations of the above types of chips.

[0201] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions / modules corresponding to the federated learning method and / or the public security information recognition method of the public security information recognition model in the embodiments of the present application. By running the non-transitory software programs, instructions, and modules stored in the memory, the processor executes various functional applications and data processing of the processor, that is, implements the federated learning method and / or the public security information recognition method of the public security information recognition model in the above method embodiments.

[0202] The memory may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created by the processor, etc. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely provided relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0203] The one or more modules are stored in the memory and, when executed by the processor, execute the federated learning method and / or the public security information recognition method of the public security information recognition model in the embodiments.

[0204] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, the memory, the receiver, and the transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to transmit and receive signals.

[0205] As an implementation, the functions of the receiver and the transmitter in this application can be considered to be implemented by a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.

[0206] As another implementation, it can be considered to use a general-purpose computer to implement the server provided in the embodiments of this application. That is, the program codes for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general-purpose processor implements the functions of the processor, the receiver and the transmitter by executing the codes in the memory.

[0207] The embodiments of this application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foregoing federated learning method and / or public security information recognition method of the public security information recognition model are implemented. The computer-readable storage medium may be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium well-known in the technical field.

[0208] Based on the foregoing embodiments of the federated learning method and / or public security information recognition method of the public security information recognition model, etc., this application also provides an embodiment of a federated learning system. The federated learning system specifically includes a server and various client devices communicatively connected to the server; the client devices are used for the federated learning method of the public security information recognition model and / or the public security information recognition method.

[0209] See Figure 6 , the server specifically includes the following:

[0210] A local data receiving module 01, configured to receive the target public security information recognition local models and respective target fine-grained local prototypes of the current round respectively sent by each of the client devices;

[0211] A fine-grained global prototype aggregation module 02, configured to aggregate all the target fine-grained local prototypes received in the current round to obtain respective fine-grained global prototypes of each fine-grained class under each class in the current round;

[0212] A feature extractor aggregation module 03, configured to aggregate the local feature extractors in all the target public security information recognition local models received in the current round to obtain the global feature extractor of the current round;

[0213] The adaptive class-level classifier aggregation module 04 is used to adaptively aggregate the local classifiers in the local model for all the target public security information received in the current round to obtain the global classifier in the current round;

[0214] The global data sending module 05 is used to generate the global model for public security information recognition in the current round based on the global feature extractor, the global classifier, and the aggregation matrix in the current round. If the current round is not the last round in the total number of iterative rounds, the global model for public security information recognition in the current round and the respective fine-grained global prototypes of each fine-grained class under each class in the current round are respectively sent to each of the client devices.

[0215] Those of ordinary skill in the art should understand that the various exemplary components, systems, and methods described in connection with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Specifically, whether to implement in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave on a transmission medium or a communication link.

[0216] It should be clear that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, the detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of this application.

[0217] In this application, the features described and / or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, and / or combined with the features of other embodiments or replace the features of other embodiments.

[0218] The above are only the preferred embodiments of this application and are not used to limit this application. For those skilled in the art, various changes and modifications can be made to the embodiments of this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the protection scope of this application.

Claims

1. A federated learning method for a public security information recognition model, characterized in that, it includes: Receiving the global model of public security information recognition in the previous round sent by the server, and the fine-grained global prototypes of each fine-grained class under each class in the previous round; Initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performing preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the global model of public security information recognition in the previous round, the local model of public security information recognition in the local area, and each of the fine-grained local prototypes, and performing gradient optimization on the local model of public security information recognition in the local area of the current round based on the preset complete loss function to obtain the local model of public security information recognition in the target current round and each target fine-grained local prototype; Sending the target local model of public security information recognition and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class in the current round, and then aggregates all the target local models of public security information recognition received in the current round to obtain the global model of public security information recognition in the current round; The global model of public security information recognition includes a global feature extractor, a global classifier, and an aggregation matrix connected in sequence; the local model of public security information recognition includes a local feature extractor, a local classifier, and an aggregation matrix connected in sequence; Correspondingly, the initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performing preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the global model of public security information recognition in the previous round, the local model of public security information recognition in the local area, and each of the fine-grained local prototypes includes: Extracting sample features corresponding to each local monitoring image sample based on the local feature extractor, and respectively allocating each of the monitoring image samples to each fine-grained class in the local area according to the sample features; Initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round; Performing hierarchical prototype contrast learning on the fine-grained local prototypes of the local current round using the sample features corresponding to each local monitoring image sample to calculate the hierarchical prototype contrast learning loss; Obtaining corresponding batch prototypes according to the sample features corresponding to each local monitoring image sample, and performing regularization processing on the batch prototypes according to the preset public security information recognition label, the global classifier, and the aggregation matrix to calculate the batch prototype regularization loss; Based on the global public security information recognition model and the local public security information recognition model of the previous round, respectively obtain the prediction probabilities corresponding to each local monitoring image sample, and calculate the global information distillation loss based on the prediction probabilities respectively output by the global public security information recognition model and the local public security information recognition model; And, calculate the debiased classifier learning loss according to the prediction probability output by the local public security information recognition model and the preset public security information recognition label; The hierarchical prototype contrast learning is performed on the fine-grained local prototypes of the current local round by using the sample features corresponding to each local monitoring image sample to calculate the hierarchical prototype contrast learning loss, including: The hierarchical prototype contrastive learning loss L is calculated based on the following formula (1) hpc : Wherein, In the above formulas (1) to (5), N is the sum of important factors; is a sample feature; D k is a local dataset composed of local monitoring image samples; I c,t is an important factor; is a sample feature optimization parameter; Ψ pos is the positive pair corresponding to the monitoring image sample; α is the feature hierarchy distance adjustment parameter; Ψ pneg is the pseudo-negative pair corresponding to the monitoring image sample; Ψ neg is the negative pair corresponding to the monitoring image sample; is the fine-grained local prototype corresponding to the t-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the j-th class.

2. The federated learning method of the public security information recognition model according to claim 1, Characterized in that, The gradient optimization of the local public security information recognition local model of the current round based on the preset complete loss function to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype includes: Determine the value of the overfitting prevention parameter of the current round based on the number of the current round in the total number of iteration rounds and the number of the total iteration rounds; Solve the complete loss function according to the value of the overfitting prevention parameter of the current round, the preset hyperparameters, the hierarchical prototype contrast learning loss, the batch prototype regularization loss, the global information distillation loss and the debiased classifier learning loss, so as to perform gradient optimization on the local public security information recognition local model of the current round based on this complete loss function, and then obtain the target public security information recognition local model of the current round and each target fine-grained local prototype.

3. The federated learning method of the public security information recognition model according to claim 1, Characterized in that, The process that the server aggregates all the target public security information recognition local models received in the current round to obtain the global public security information recognition model of the current round includes: The server aggregates the local feature extractors in all the target public security information recognition local models received in the current round to obtain the global feature extractor of the current round; And, the server performs adaptive class-level classifier aggregation on the local classifiers in all the target public security information recognition local models received in the current round to obtain the global classifier of the current round; The server generates the global public security information recognition model of the current round based on the global feature extractor, the global classifier and the aggregation matrix of the current round.

4. A public security information recognition method, Characterized in that, Includes: Obtain the monitoring image data of the target area; Input the monitoring image data into the global public security information recognition model obtained in advance based on the federated learning method of the public security information recognition model according to any one of claims 1 to 3, so as to determine the public security information recognition result of the monitoring image data according to the output of this global public security information recognition model.

5. A federated learning device for a public security information recognition model, It is characterized in that including a global data receiving module, configured to receive the public security information recognition global model of the previous round sent by the server and the fine-grained global prototypes of each fine-grained class under each class of the previous round a model training module, configured to initialize the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and perform preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the public security information recognition global model of the previous round, the public security information recognition local model of the local, and each of the fine-grained local prototypes, and perform gradient optimization on the public security information recognition local model of the local in the current round based on the preset complete loss function to obtain the target public security information recognition local model of the current round and each target fine-grained local prototype a local data sending module, configured to send the target public security information recognition local model and each of the target fine-grained local prototypes to the server, so that the server aggregates all the target fine-grained local prototypes received in the current round to obtain the fine-grained global prototypes of each fine-grained class under each class of the current round, and then aggregates all the target public security information recognition local models received in the current round to obtain the public security information recognition global model of the current round wherein, the public security information recognition global model includes a global feature extractor, a global classifier, and an aggregation matrix connected in sequence; the public security information recognition local model includes a local feature extractor, a local classifier, and an aggregation matrix connected in sequence Correspondingly, the initializing the fine-grained local prototypes of the local current round based on the fine-grained global prototypes of the previous round, and performing preset hierarchical prototype contrast learning, batch prototype regularization, and global information distillation processing according to each local monitoring image sample, the preset public security information recognition label, the public security information recognition global model of the previous round, the public security information recognition local model of the local, and each of the fine-grained local prototypes includes extracting sample features corresponding to each local monitoring image sample based on the local feature extractor, and respectively assigning each of the monitoring image samples to each fine-grained class of the local according to the sample features initializing the fine-grained local prototypes of the local current round according to the fine-grained global prototypes of the previous round performing hierarchical prototype contrast learning on the fine-grained local prototypes of the local current round by using the sample features corresponding to each local monitoring image sample to calculate the hierarchical prototype contrast learning loss obtaining corresponding batch prototypes according to the sample features corresponding to each local monitoring image sample, and performing regularization processing on the batch prototypes according to the preset public security information recognition label, the global classifier, and the aggregation matrix to calculate the batch prototype regularization loss Based on the global model for public security information recognition and the local model for public security information recognition in the previous round, obtain the prediction probabilities corresponding to each local surveillance image sample respectively, and calculate the global information distillation loss based on the prediction probabilities respectively output by the global model for public security information recognition and the local model for public security information recognition; Moreover, calculate the debiased classifier learning loss according to the prediction probability output by the local model for public security information recognition and the preset public security information recognition label; The hierarchical prototype contrast learning for calculating the hierarchical prototype contrast learning loss by using the sample features corresponding to each local surveillance image sample in the local current round for the fine-grained local prototype includes: The hierarchical prototype contrastive learning loss L is calculated based on the following formula (1) hpc :[[]]END]] Wherein, In the above formulas (1) to (5), N is the sum of important factors; is a sample feature; D k is a local dataset composed of local monitoring image samples; I c,t is an important factor; is a sample feature optimization parameter; Ψ pos is the positive pair corresponding to the monitoring image sample; α is the feature level distance adjustment parameter; Ψ pneg is the pseudo-negative pair corresponding to the monitoring image sample; Ψ neg is the negative pair corresponding to the monitoring image sample; is the fine-grained local prototype corresponding to the t-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the c-th class; is the fine-grained local prototype corresponding to the m-th fine-grained category under the j-th class.

6. An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, Characterized in that, When the processor executes the computer program, it implements the federated learning method of the public security information recognition model as described in claims 1 to 3, and / or implements the public security information recognition method as described in claim 4.

7. A computer-readable storage medium, on which a computer program is stored, Characterized in that, When the computer program is executed by the processor, it implements the federated learning method of the public security information recognition model as described in claims 1 to 3, and / or implements the public security information recognition method as described in claim 4.

8. A federated learning system, Characterized in that, It includes: A server, and each client device communicatively connected to the server; The client device is used to execute the federated learning method of the public security information recognition model as described in claims 1 to 3, and / or is used to execute the public security information recognition method as described in claim 4; The server includes: A local data receiving module, configured to receive the target local model for public security information recognition in the current round and each target fine-grained local prototype respectively sent by each client device; A fine-grained global prototype aggregation module, configured to aggregate all the target fine-grained local prototypes received in the current round to obtain the respective fine-grained global prototypes of each fine-grained class under each class in the current round; A feature extractor aggregation module, configured to aggregate the local feature extractors in all the target local models for public security information recognition received in the current round to obtain the global feature extractor in the current round; An adaptive class-level classifier aggregation module, configured to perform adaptive class-level classifier aggregation on the local classifiers in all the target local models for public security information recognition received in the current round to obtain the global classifier in the current round; A global data sending module, configured to generate the global model for public security information recognition in the current round based on the global feature extractor, global classifier, and aggregation matrix in the current round. If the current round is not the last round in the total number of iterative rounds, send the global model for public security information recognition in the current round and the respective fine-grained global prototypes of each fine-grained class under each class in the current round to each client device respectively.

Images