Game-based distributed learning method for crowdsourced image recognition on mobile terminals

Abstract

The present invention proposes a game-based distributed learning method for crowdsourced image recognition on mobile terminals. The method comprises: a central server constructs a multi-person cooperative game model based on mobile terminal data transfer; mobile terminals define their own training image data offset metrics and minimum data acquisition benefit values; mobile terminals perform distributed computations on image data allocation to obtain an optimized computational target; obtain the optimal Lagrange multiplier and data movement strategy; optimize and solve the problem using a decentralized tuning method; and use the obtained optimal strategy to move image data, which is then transmitted point-to-point between mobile terminals via WiFi or a cellular network. The method effectively balances data distribution between mobile terminals in different decentralized environments and ensures fairness in distributed learning training, without sacrificing the performance of a few mobile terminals to improve the overall system performance, thereby encouraging mobile terminals to participate in distributed learning.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and in particular to a game-based distributed learning method for crowdsourcing recognition of mobile terminal images. Background Art

[0002] Currently, crowdsourcing-based mobile terminal image recognition is gaining widespread research and application. By placing proprietary image recognition models on mobile terminals, it leverages decentralized human resources and mobile terminal computing resources to provide users with personalized services. For example, NavNet is an edge computing-based high-precision map construction and positioning solution for autonomous driving. Using local cameras, it applies deep learning models to recognize images and achieve semantic 3D reconstruction of road scenes. It also uses crowdsourcing to update high-precision maps. The FedVision platform, proposed by Nanyang Technological University, provides a security surveillance solution for smart cities. It also utilizes a large number of edge-based image labeling and local model training methods. In these applications, a large number of mobile terminals participate in model training by providing local images, improving both the accuracy of the global image recognition model and the performance of local personalized image recognition. Compared to centralized training methods, crowdsourcing-based training involves mobile terminal devices connected via WiFi or cellular networks. This decentralized collaboration effectively reduces the cost of training image recognition models.

[0003] Currently, the main technical challenge facing distributed learning for mobile terminal image recognition is that the data on the participating mobile terminals is likely to be non-independent and identically distributed (Non-IID), that is, the categories and quantities of local image data on different mobile terminals vary greatly. This heterogeneity of image data will significantly reduce the performance of distributed learning to generate image recognition models. Existing research has been unable to solve the problem of distributed learning for mobile terminal image recognition.

[0004] For example, Chinese patent application number 202111419506.2 discloses an image recognition method based on improved PATE, which includes:

[0005] Based on the federated learning algorithm, each client uses its own local image dataset to train the parameters of its own teacher model, and the central server aggregates and updates the parameters of each teacher model in each iteration of the training;

[0006] Get an unlabeled image dataset;

[0007] Using the trained teacher models and combining the aggregation mechanism, the unlabeled image dataset is predicted to obtain a labeled image dataset;

[0008] Training a student model using the labeled training image dataset;

[0009] The student model is used to perform image recognition on the image data to be identified. Multiple teacher models are trained through distributed learning. A central server aggregates the models using a model aggregation mechanism, and then guides the student model to predict images. However, this method does not consider the situation in real-world scenarios where large data offsets across distributed learning nodes can lead to significant differences in model parameters.

[0010] Based on the above-mentioned technical problems existing in the prior art, the present invention provides a game-based distributed learning method for crowdsourcing recognition of mobile terminal images. Summary of the Invention

[0011] The present invention proposes a game-based distributed learning method for crowdsourcing recognition of mobile terminal images.

[0012] The present invention adopts the following technical solutions:

[0013] A game-based distributed learning method for crowdsourced image recognition on mobile terminals, comprising:

[0014] Step 1: The central server builds a multi-person cooperative game model based on mobile terminal data transfer;

[0015] Step 2: The mobile terminal defines its own training image data offset measurement value and minimum data acquisition benefit value;

[0016] Step 3: The mobile terminals distribute the image data, the central server establishes an optimization problem model, and the mobile terminals construct a Lagrangian function for the optimization problem to obtain the optimization calculation target.

[0017] Step 4: The mobile terminal obtains the optimal Lagrange multiplier and data movement strategy through a subgradient descent algorithm;

[0018] Step 5: The mobile terminal applies a decentralized tuning method for large-scale dynamic networks to optimize the solution process;

[0019] In step 6, the mobile terminal moves the image data according to the obtained optimal strategy, and the image data is transmitted point-to-point between the mobile terminals via WiFi or cellular network.

[0020] Furthermore, in step 2, the training image data offset metric value of the mobile terminal is defined as the first utility The minimum data acquisition benefit value is defined as the second utility The benefit value is fixed under a fixed initial data distribution, and the training data on the mobile terminal is defined as an N-dimensional array s i =(s1,s2,…,s N), the data offset of the mobile terminal is measured using cosine similarity, using unit vector For reference, the training image data offset metric and the minimum data acquisition benefit value of the mobile terminal are calculated by the following formula:

[0021]

[0022] Among them, k means the data category, s i,k Represents the number of k-th type of data on mobile terminal i.

[0023] Furthermore, in step 3, the central server establishes an optimization problem model based on the Nash equilibrium solution, and the optimization problem model is calculated using the following formula:

[0024]

[0025] Among them, T k represents the global total number of data of the kth category. The constraints are valid for all i∈[1,N] and k∈[1,K]. The range of the cosine function is [-1,1], and for any values ​​of i and k, s i,k >0, so U i ∈(0,1], in U i ∈(0,1], the optimization objective is -log(U i )>0,d i A predetermined minimum benefit is obtained for each mobile terminal.

[0026] Furthermore, in step 3, the mobile terminal constructs a Lagrangian function for the optimization problem, and the obtained optimization calculation target is:

[0027]

[0028] The solution is obtained separately for each mobile terminal, and λ is the Lagrange multiplier.

[0029] Furthermore, step 4 includes:

[0030] Step 4.1, execute in parallel on all mobile terminals: solve turn up is the optimal data distribution method for the current iteration;

[0031] Step 4.2, calculate the local subgradient

[0032] Step 4.3: interact with adjacent mobile terminals and collect their sub-gradient information. Update the local Lagrange multiplier based on the sub-gradient information of the adjacent mobile terminals.

[0033] Step 4.4: Repeat steps 4.1 to 4.3 until λ converges.

[0034] Furthermore, in step 1, the central server and the mobile terminal, and different mobile terminals communicate via WiFi or 5G cellular network.

[0035] Furthermore, in step 1, the central server is used to maintain the metadata of the terminal, the mobile terminal is used to train the model, and the image data or related information required for model upgrades are exchanged in a point-to-point manner through communication between the mobile terminals.

[0036] Furthermore, in step 4.3, the information collected by the mobile terminal from other mobile terminals includes two message types: RequestSub and SendSub. RequestSub and SendSub are sent via a base station or WiFi. RequestSub is used to request from a mobile terminal j, and SendSub is used to receive a sub-gradient from a specific mobile terminal j.

[0037] Furthermore, step 5 includes:

[0038] Step 5.1: Limit the number of mobile terminals that can communicate, restricting the interaction of a mobile terminal in each iteration to only its neighboring mobile terminals. Set a timeout threshold on the mobile terminal so that mobile terminals that exit the system or cannot respond due to network connection can be silently removed without disrupting system performance.

[0039] Step 5.2, remove the communication with the mobile terminal that responds slower than the set time threshold. In a highly dynamic network scenario, set a timeout threshold on the mobile terminal so that the mobile terminal that exits the system or cannot respond due to network connection can be silently removed without interfering with system performance.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] 1. The game-based distributed learning method for crowdsourced image recognition on mobile terminals described in this invention effectively balances data distribution among mobile terminals in different decentralized environments and ensures fairness in distributed learning training. It does not sacrifice the performance of a few mobile terminals to improve the overall performance of the system, thereby encouraging mobile terminals to participate in distributed learning.

[0042] 2. The game-based distributed learning method for crowdsourced image recognition on mobile terminals described in the present invention can limit the number of mobile terminals with which a mobile terminal interacts in each iteration, greatly reducing the number of communications required during the method iteration process. Compared with the traditional decentralized distributed learning scheme that improves distributed learning performance by increasing the total number of mobile terminals and the number of neighbors, it is more convenient and economical, and is more suitable for actual situations where communication costs are high and computing power and resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of a game-based distributed learning system for crowdsourcing recognition of mobile terminal images in an embodiment of the present invention;

[0044] Figure 2 3 is a comparison chart showing the impact of mobile terminal scale on decentralized learning task performance in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other unless there is a conflict.

[0046] Example

[0047] like Figure 1 As shown in the figure, a central server and multiple participating mobile terminals constitute a game-based distributed learning system for crowdsourced image recognition on mobile terminals. The central server and mobile terminals, as well as mobile terminals themselves, communicate via WiFi or 5G cellular networks. To protect privacy, the central server only maintains metadata about the mobile terminals, such as the time they join or leave the system and the amount of allocated incentives. The mobile terminals are responsible for training the models and exchanging image data or related information required for model upgrades in a point-to-point manner through inter-device communication. Since the environment has a certain degree of trust, mobile terminals in the system can share some image data with other mobile terminals, thereby improving the overall performance of the image recognition application. At the same time, in the absence of central service scheduling, the main goal of different mobile terminals is to improve their own benefits without considering the benefits of other mobile terminals. In the distributed learning process, multiple mobile terminals collaborate to solve a problem. The original data of each mobile terminal is stored locally, thereby preventing individuals from disclosing their personal data to others to protect data privacy and allowing data processing to be performed closer to the data generation location to improve latency performance.

[0048] The game-based distributed learning method for crowdsourcing recognition of mobile terminal images includes:

[0049] Step 1: The central server builds a multi-person cooperative game model based on mobile terminal data transfer;

[0050] Based on the number of mobile terminals participating in the distributed learning system and the amount of data, the central server models the data sharing problem as a multi-person cooperative game of image data transfer between multiple mobile terminals. The shareable data can be regarded as the common benefits that need to be divided. The goal of each mobile terminal is to optimize its own utility function U i If the negotiation fails, each mobile terminal obtains the predetermined minimum benefit d i , which is the efficiency obtained when no data allocation is performed, that is, in the game, all benefits will be higher than before;

[0051] Step 2: The mobile terminal defines its own training image data offset measurement value and minimum data acquisition benefit value;

[0052] The training image data deviation measurement value of the mobile terminal and the minimum data acquisition benefit value are corresponded to the utility function of the mobile terminal in the model; the utility function is divided into two parts: the training image data deviation measurement value of the mobile terminal is defined as the utility The minimum data acquisition benefit value is positioned as utility Under a fixed initial data distribution, the training data on each mobile terminal is defined as an N-dimensional array s i =(s1,s2,…,s N ), there are many ways to optimize the utility function U(s i )(Abbreviated as U i )

[0053] The data offset of the mobile terminal is measured using cosine similarity, specifically, using the unit vector For reference, the unit vector 1 K This represents an ideal balanced data distribution. Because cosine similarity is independent of vector length, the specific length of the reference vector is not considered. When calculating cosine similarity, all local image data of the mobile terminal is required, including both shareable and non-shareable parts. The training image data offset metric and the minimum data acquisition benefit value of the mobile terminal are calculated using the following formula:

[0054]

[0055] Among them, k means the data category, s i,k represents the number of k-th category data on mobile terminal i;

[0056] Step 3: The mobile terminal distributes the image data in a distributed computing manner, which specifically includes the following two steps:

[0057] In step 3.1, the central server establishes an optimization problem model. The multi-player cooperative game for transferring terminal image data described in step 1 has a Nash equilibrium solution. One of the important features of a Nash equilibrium solution is that it provides fairness in image data distribution, thereby incentivizing the participation of mobile terminals. Based on the Nash equilibrium solution, the following optimization model is established:

[0058]

[0059] Subject to:s i,k ≥0,

[0060]

[0061] Among them, T k Represents the global total number of k-th category data. The above optimization model is valid for all i∈[1,N] and k∈[1,K] in the constraints. The range of the cosine function is [-1,1], and for any values ​​of i and k, s i,k >0, so U i ∈(0,1], in this range, the optimization objective is -log(U i )>0;

[0062] In step 3.2, the mobile terminal performs distributed computing and constructs the Lagrangian function through the optimization problem. The optimization calculation target is:

[0063]

[0064] Among them, is solved separately on each mobile terminal, where λ is the Lagrange multiplier;

[0065] Step 4: The mobile terminal uses a subgradient descent algorithm to find the optimal Lagrange multiplier and data movement strategy. Specifically, it is divided into the following four steps:

[0066] Step 4.1, by solving turn up This step is performed in parallel on all mobile terminals;

[0067] Step 4.2, calculate the local subgradient

[0068] Step 4.3: Interact with neighboring mobile terminals and collect their sub-gradient information, and update the local Lagrange multiplier accordingly.

[0069] Step 4.4, repeat steps 4.1 to 4.3 until λ converges;

[0070] In step 4.3, the mobile terminal collects information from other mobile terminals in the system. Therefore, the mobile terminal supports two types of messages: RequestSub and SendSub. These two messages are sent via the base station or WiFi and are used to request and receive sub-gradients from a specific mobile terminal j, respectively. Once a mobile terminal completes the optimization work in step 4.1, the mobile terminal will send a RequestSub message to other mobile terminals. If mobile terminal j receives this message and prepares the requested sub-gradient, it will send it back in a SendSub message. The above sub-gradient is stored in the mobile terminal's memory. The memory stores the sub-gradient value of each round. Different versions of sub-gradients are managed using the distributed consensus algorithm Raft, allowing mobile terminals to exit or rejoin at any time to catch up with the calculation progress.

[0071] Step 5: The mobile terminal applies a decentralized optimization method for large-scale dynamic networks to optimize the solution process. Specifically, it includes the following two steps:

[0072] Step 5.1: Limit the number of communicable mobile terminals. Limit the interaction of a mobile terminal in each iteration to only its adjacent mobile terminals, and not to interact with remote mobile terminals. Depending on the convergence requirements, you can choose to use the nhop neighborhood instead of only the direct neighborhood.

[0073] Step 5.2: Remove the communication with the mobile terminal whose response time exceeds the set threshold. In highly dynamic network scenarios, a timeout threshold can be set on the mobile terminal so that the mobile terminal that exits the system or cannot respond due to network connection during this stage can be silently removed without interfering with system performance.

[0074] In step 6, the mobile terminal moves the image data according to the obtained optimal strategy, and the image data is transmitted point-to-point between the mobile terminals via WiFi or cellular network.

[0075] Combine Figure 1 In the actual edge-side distributed learning training scenario shown, the game-based distributed learning system for crowdsourced mobile terminal image recognition is deployed and run according to the following steps:

[0076] S1, the central server builds a multi-person cooperative game model based on mobile terminal data transfer;

[0077] S2, the mobile terminal defines its own training image data offset measurement value and minimum data acquisition benefit value;

[0078] S3, distributed computing image data distribution on mobile terminals;

[0079] S3.1, the central server establishes an optimization problem model;

[0080] S3.2, the mobile terminal performs distributed computing, constructs a Lagrangian function through the optimization problem, and obtains the optimization calculation target;

[0081] S4, the mobile terminal obtains the optimal Lagrange multiplier and data movement strategy through the subgradient descent algorithm;

[0082] S4.1, solve the optimal data distribution method under the current iteration;

[0083] S4.2, calculate local subgradient;

[0084] S4.3, interact with neighboring mobile terminals and collect their sub-gradient information, and update the local Lagrange multiplier accordingly;

[0085] S4.4, repeat S4.1-S4.3 until the local Lagrange multiplier converges;

[0086] S5, the optimization solution process of decentralized tuning methods for large-scale dynamic networks applied on mobile terminals;

[0087] S5.1, a mobile terminal shall limit its interactions in each iteration to only its neighboring mobile terminals;

[0088] S5.2, remove communications with mobile terminals whose response time exceeds a certain threshold;

[0089] S6, the mobile terminal moves the image data according to the obtained optimal strategy, and the image data is transmitted point-to-point between the mobile terminals via WiFi or cellular network;

[0090] S7, simulation platform evaluation:

[0091] In the experiment, this embodiment used 20 mobile terminals and executed the algorithm with a fixed number of iterations. The local dataset construction method is divided into two parts: constructing an IID dataset: randomly shuffling and assigning a portion of the original dataset to each mobile terminal; constructing a non-IID dataset: arranging the data by label, randomly selecting class c and setting it as the main category, accounting for a certain percentage of the total training data (the percentage is q), and randomly selecting data from the remaining categories and evenly assigning them to the mobile terminals. In this embodiment, the parameters c of the dataset to be optimized are set to 10 and q is set to 35% as a comparison of the optimization effect;

[0092] Secondly, a convolutional neural network is established. The model uses a convolutional neural network model with two convolutional layers and two fully connected layers. The two convolutional layers have 32 and 64 output channels respectively. The kernel size of both layers is 3x3, and the stride is 1. These two layers are followed by a Relu layer and a dropout layer. The first fully connected layer has 128 units activated by ReLu, and the second fully connected layer has an output size of 62, followed by a Softmax output layer. During training, the loss function is categorical cross entropy, and the model accuracy metric is top-1 accuracy. In this embodiment, a small amount of data is used to simulate the scenario on the mobile terminal. Therefore, for local training of each customer, the batch parameter is set to 8, each local Epoch is set to 3, and the Adam optimizer is used with a learning rate of 0.001 and a coefficient of (0.9, 0.999).

[0093] Figure 2 The method FairDist described in this embodiment is compared with the existing decentralized training methods D-PSGD and PENS, and the centralized data enhancement training method Hybrid-FL. Figure 2 It can be seen that under all different total number of mobile terminal settings, the method described in this embodiment achieves better accuracy than the existing method on the original data set. When using the same epoch (a concept of deep learning model training, an epoch means that the neural network model traverses all samples once), this embodiment also compares the different final average accuracies that can be achieved when the total number of mobile terminals increases. It can be seen that when the total number of mobile terminals is increased, the average accuracy generally increases. In the case of Non-IID decentralized distributed learning, increasing the total number of mobile terminals can improve performance. It is preliminarily speculated that due to the large number of mobile terminals, all mobile terminals contain more types of data distributions. Therefore, during the mobile terminal communication process, more "experience" can be obtained from other mobile terminals to improve the performance of the test set.

[0094] This experiment set up 1, 2, 3 and 4 neighbors for experiments respectively. Under the Non-IID experimental setting, the algorithm proposed in this embodiment still achieved better performance. According to the final average accuracy results obtained by various methods under different experimental conditions, it can be seen that increasing the number of neighbors can improve the performance of distributed learning. In short, under the Non-IID dataset setting, increasing the total number of mobile terminals and the number of neighbors can improve the performance of decentralized distributed learning. Under different experimental environments, the method described in this embodiment achieved better performance.

[0095] The present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims.

Claims

1. A game-based distributed learning method for crowdsourcing image recognition on mobile terminals, characterized by: include: Step 1: The central server builds a multi-person cooperative game model based on mobile terminal data transfer; Step 2: The mobile terminal defines its own training image data offset metric value and the minimum data acquisition benefit value. The training image data offset metric value of the mobile terminal is defined as the first utility The minimum data acquisition benefit value is defined as the second utility The benefit value is fixed under a fixed initial data distribution, and the training data on the mobile terminal is defined as an N-dimensional array s i =(s1,s2,…,s N ), the data offset of the mobile terminal is measured using cosine similarity, using unit vector For reference, the training image data offset metric and the minimum data acquisition benefit value of the mobile terminal are calculated by the following formula: Where k means the data category, s i,k represents the number of k-th category data on mobile terminal i; Step 3: The mobile terminals distribute the image data, and the central server establishes an optimization problem model. The mobile terminals construct a Lagrangian function for the optimization problem and obtain the optimization calculation target. The optimization problem model established by the central server is based on the Nash equilibrium solution. The optimization problem model is calculated using the following formula: Among them, T k represents the global total number of data of the kth category. The constraints are valid for all i∈[1,N] and k∈[1,K]. The range of the cosine function is [-1,1], and for any values ​​of i and k, s i,k >0, so U i ∈(0,1], in U i ∈(0,1], the optimization objective is -log(U i )>0,d i Obtain a predetermined minimum benefit for each mobile terminal; Step 4: The mobile terminal uses the sub-gradient descent algorithm to find the optimal Lagrangian multiplier and data movement strategy. The mobile terminal constructs a Lagrangian function for the optimization problem, and the optimization calculation target is: Here, the solution is obtained separately on each mobile terminal, and λ is the Lagrange multiplier; Step 5: The mobile terminal applies a decentralized tuning method for large-scale dynamic networks to optimize the solution process; In step 6, the mobile terminal moves the image data according to the obtained optimal strategy, and the image data is transmitted point-to-point between the mobile terminals via WiFi or cellular network.

2. The game-based distributed learning method for crowdsourcing recognition of mobile terminal images according to claim 1, characterized in that: Step 4 includes: Step 4.1, execute in parallel on all mobile terminals: solve turn up is the optimal data distribution method for the current iteration; Step 4.2, calculate the local subgradient Step 4.3: interact with adjacent mobile terminals and collect their sub-gradient information. Update the local Lagrange multiplier based on the sub-gradient information of the adjacent mobile terminals. Step 4.4: Repeat steps 4.1 to 4.3 until λ converges.

3. The game-based distributed learning method for crowdsourcing recognition of mobile terminal images according to claim 1, characterized in that: In step 1, the central server and the mobile terminal, as well as different mobile terminals, communicate via WiFi or 5G cellular network.

4. The game-based distributed learning method for crowdsourcing recognition of mobile terminal images according to claim 1, characterized in that: In step 1, the central server is used to maintain the metadata of the terminal, the mobile terminal is used to train the model, and the image data or related information required for model upgrades are exchanged in a point-to-point manner through communication between the mobile terminals.

5. The game-based distributed learning method for crowdsourcing recognition of mobile terminal images according to claim 2, characterized in that: In step 4.3, the information collected by the mobile terminal from other mobile terminals includes two message types: RequestSub and SendSub. RequestSub and SendSub are sent through the base station or WiFi. RequestSub is used to request from a mobile terminal j, and SendSub is used to receive sub-gradients from a specific mobile terminal j.

6. The game-based distributed learning method for crowdsourcing recognition of mobile terminal images according to claim 1, characterized in that: Step 5 includes: Step 5.1: Limit the number of mobile terminals that can communicate, restricting the interaction of a mobile terminal in each iteration to only its neighboring mobile terminals. Set a timeout threshold on the mobile terminal so that mobile terminals that exit the system or cannot respond due to network connection can be silently removed without disrupting system performance. Step 5.2, remove the communication with the mobile terminal that responds slower than the set time threshold. In a highly dynamic network scenario, set a timeout threshold on the mobile terminal so that the mobile terminal that exits the system or cannot respond due to network connection can be silently removed without interfering with system performance.

Images