A visual positioning method and device based on semi-supervised federated learning and a medium

Abstract

The application discloses a visual positioning method and device based on semi-supervised federated learning and a medium. The method comprises the following steps: after a visual scene changes, a user collects untagged image data by using a client device; in each communication round, the client under each edge server is trained based on the untagged data, and a model parameter decomposition strategy is adopted to distinguish supervised learning from unsupervised learning; the local client is trained for multiple rounds based on an unsupervised loss function which is fused with pseudo-labels and consistency regularization loss, and updated model parameters are uploaded to the edge server end; in each communication round, the edge server end comprehensively considers the influence of the tagged data and the untagged data, and model parameters are aggregated according to an aggregation algorithm HierAgg; multiple communication rounds are performed, and an updated model is obtained. The application solves the problem of adaptive updating of the model along with the change of the scene based on semi-supervised federated learning by using the untagged data generated after the change of the visual scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of visual positioning, and particularly relates to a visual positioning method, device and medium based on semi-supervised federated learning. BACKGROUND

[0002] Visual positioning is an image-based positioning method, which is applied to the fields of automatic driving, augmented reality, robot navigation and the like. A visual positioning method based on a convolutional neural network performs positioning by training an end-to-end neural network, which is essentially a regression model. A user inputs a picture into the model, and the model directly regresses the position information of the picture. However, when the visual scene changes, the image features in the scene will also change, and the pre-constructed model cannot cope with the change. At this time, the positioning error of the image input by the user is often large. Scene change is a high-frequency occurrence in the real world, but the model adaptation problem under scene change is rarely considered in the past work.

[0003] If a centralized training method is used for model updating, a data privacy problem will occur. The user needs to upload the photographed picture data to the central server, and the image data contains a lot of private information, which has a privacy security problem. Federated learning is a distributed machine learning framework, which is different from the traditional centralized training. It limits the training process to the local client of the user, which avoids the security problem of directly uploading data by the user and better protects the data privacy in the model training.

[0004] When the scene changes, the newly appearing visual features can be regarded as unlabeled data. Labeling these unlabeled data will consume a lot of manpower and material resources. Directly using the unlabeled data for model training is a more simple and efficient way. Semi-supervised federated learning integrates semi-supervised learning technology into the federated learning framework. On the one hand, federated learning can protect the data privacy in the model training process, and on the other hand, semi-supervised learning technology can use unlabeled data for model updating. Combining semi-supervised federated learning with visual positioning can effectively solve the model updating problem of visual positioning and the difficult problem of visual positioning in the scene change environment. SUMMARY

[0005] The application aims at the problem that the existing visual positioning method cannot adaptively update the model with the change of the scene. The application provides a visual positioning method, device and medium based on semi-supervised federated learning.

[0006] Technical scheme: The visual positioning method based on semi-supervised federated learning provided by the application adopts a layered architecture of cloud server-edge server-client, and aggregates in the edge server and the cloud server respectively, so that the edge server and the cloud server are both updated. The specific implementation process is as follows:

[0007] When the visual scene changes, the user uses the client device to collect unlabelled image data;

[0008] In each communication round, the clients under each edge server train based on the unlabelled data, and adopt a model parameter decomposition strategy to distinguish between supervised learning and unsupervised learning; the pseudo label and consistency regularization loss are fused in the unsupervised loss function; the local client performs multi-round training based on the loss, and uploads the updated model parameters to the edge server end;

[0009] In each communication round, the edge server end comprehensively considers the influence of labelled data and unlabelled data, and performs model parameter aggregation according to the aggregation algorithm HierAgg;

[0010] After multiple communication rounds, an updated visual positioning model is obtained; the client uses the updated model for positioning.

[0011] Further, the supervised learning implementation process is as follows:

[0012] The model parameters ω are decomposed into two variables, σ for supervised learning and ψ for unsupervised learning, i.e. ω = σ + ψ;

[0013] For learning of labelled data, standard supervised learning is performed on σ by minimizing the labelled loss, and ψ remains unchanged during the training process; the loss function of the edge server end model training is defined as follows:

[0014]

[0015] where f(x j ; σ s ) represents the model σ S of the edge server for the model prediction of the labelled sample x j ; l(.) represents the loss function of the PoseNet model for each sample.

[0016] Further, the unsupervised learning implementation process is as follows:

[0017] Unsupervised learning is performed on the model parameters ψ, σ is fixed during the learning stage, and is minimized by minimizing the unlabelled loss term; at the client, the unsupervised loss function is improved by combining consistency regularization and pseudo label method;

[0018] Define the edge server as k, and there are multiple clients under the server, defined as i ∈ {1, …, N}; for the image data x input by the user, use the image data enhancement method to obtain x and are input into the i-th client model for forward inference to obtain the predicted value f(x; ωi ) and ω i denotes the model parameter of the i-th client; the pseudo label is calculated as:

[0019]

[0020] where λ represents the weight factor of the predicted value; for the i-th client model, the loss function of the pseudo label training is:

[0021]

[0022] where j denotes the unlabeled sample, |D i | is the number of unlabeled samples on the i-th client, l(.) is the loss function of the positioning model for each sample j, y j denotes the calculated pseudo label;

[0023] The goal of consistency regularization is to make the model predicted value f(u; ω i ) of the sample u as small as possible with the distance value between the model predicted value of the perturbed sample , where the distance measurement uses the KL divergence; based on the KL divergence, the consistency regularization loss of the i-th client is:

[0024]

[0025] where |D i | is the number of unlabeled samples on the i-th client;

[0026] The unsupervised loss of the i-th client is:

[0027]

[0028]

[0029] where L PL denotes the pseudo label loss, L CR denotes the consistency regularization loss, are the L2 and L1 regularization item coefficients respectively; μ t is a dynamic coefficient that changes with each round of communication, T is the number of communication rounds of the i-th client model training, t represents the current training round, μ t balances the influence of the pseudo label on unsupervised training.

[0030] Further, the model parameter aggregation implementation process according to the aggregation algorithm HierAgg is as follows:

[0031] After T1 local updates at each client, each edge server aggregates the model parameters from the clients; after every T2 edge model aggregations, the cloud server aggregates all edge servers' models; the aggregation strategy between edge and cloud uses the FedAvg algorithm, and the aggregation formula between edge and client is as follows:

[0032]

[0033] wherein, denotes the aggregated model parameters, t denotes the communication round, denotes the model parameters trained by the i-th client based on the unlabeled dataset D i , t denotes the model trained by the edge server k on the labeled dataset D k , and N denotes the number of clients. Further, the visual positioning model is a PoseNet model.

[0034] The device of the application comprises a memory and a processor, wherein:

[0035] The memory is used for storing a computer program capable of running on the processor;

[0036] The processor is used for executing the steps of the visual positioning method based on semi-supervised federated learning as described above when the computer program is running.

[0037] The storage medium of the application stores a computer program, and the computer program is executed by at least one processor to implement the steps of the visual positioning method based on semi-supervised federated learning as described above.

[0038] Advantages: Compared with the prior art, the advantages of the application are that the introduction of pseudo-labels and consistency regularization loss enables the positioning model to utilize unlabeled data for federated learning in a semi-supervised manner, thereby achieving dynamic updating and adaptation in the case of scene changes; compared with traditional methods, the application can effectively maintain positioning accuracy and avoid the problem of accuracy decline caused by scene changes, thereby improving the robustness of visual positioning and making the positioning effect more stable; and since federated learning is a distributed machine learning framework, it can effectively protect data privacy during model training and provide better data security for the training process.

[0039] BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The figure is a schematic diagram of the principle of the application;

[0041] Figure 2 The figure is an effect diagram of the application on an outdoor dataset.​ DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the present invention proposes a visual positioning method based on semi-supervised federated learning, which adopts a federated learning framework for model training. The client locally stores unlabeled data, which represents the data generated after the scene changes. The client device performs unsupervised training based on this data. The pseudo-label method and consistency regularization are used in the training process, and then the updated model parameters are uploaded to the server for aggregation. In the model aggregation process, based on a cloud-edge-client layered architecture used in the present invention, a layered aggregation strategy is proposed, which performs aggregation at the edge and cloud respectively, so that the edge and cloud models can be updated. After multiple rounds of communication, the updated model is finally obtained. The key problem solved by the present invention is how the visual positioning model adaptively updates the model when the visual scene changes.

[0044] The PoseNet model, an end-to-end localization method, was used for visual localization. This method directly generates position information by inputting an image into the model. The model was trained using a dataset containing RGB images and the pose information corresponding to each image. The model's primary function is to estimate the six-degree-of-freedom pose information of the RGB image, namely the image's position and orientation coordinates.

[0045] In this paper, a novel federated learning architecture is applied to visual localization tasks, combining the advantages of cloud computing and edge computing. Traditionally, research on federated learning (FL) has focused on scenarios where cloud servers serve as parameter servers. However, with the development of edge computing technology, edge server-based FL systems have received increasing attention. The main advantage of cloud servers is their ability to process large amounts of training data, while edge servers enable fast communication with local clients, accelerating model updates. This paper aims to combine these two advantages for visual localization through a layered FL architecture that collaborates with the cloud, edge, and client. Compared to pure cloud-based FL, this layered architecture significantly reduces communication costs with the cloud while leveraging the efficiency of edge computing to enable rapid model updates with the client. This significantly shortens model training time and reduces the number of local iterations. Furthermore, since cloud servers can access a wider range of datasets, it also facilitates the scalability of visual localization systems across multiple scenarios. When new scenarios need to be added, simply add the corresponding edge servers.

[0046] The layered FL architecture consists of three layers: cloud servers, edge servers, and user devices. The top layer of this system consists of cloud servers, primarily responsible for storing and managing global models. The middle layer consists of edge servers, one for each specific scenario, which primarily store scenario-specific positioning models. The bottom layer consists of user devices, which users use to access indoor positioning services.

[0047] To address the impact of scene changes on the accuracy of visual localization models, this paper proposes a semi-supervised federated learning framework to handle scene updates. When the scene changes, newly generated visual features are treated as unlabeled data. This paper treats the model update process as a semi-supervised learning problem and handles it within a federated learning framework. This approach not only helps protect data privacy but also effectively adapts to scene changes while reducing communication overhead, thereby improving the accuracy of the visual localization model.

[0048] The semi-supervised federated learning framework proposed in this paper combines two semi-supervised learning methods, pseudo-labeling and consistency regularization, to utilize the unlabeled data of each client. In this framework, the edge server is responsible for storing labeled data, while the client is responsible for storing unlabeled data. This setting is consistent with the scenario where the label is on the server, which is a more challenging situation. In this scenario, assume there is an edge server labeled S and N clients. The server keeps a limited labeled data set D S ={(x1,y1),…,(x n ,y n )}, and each client has an unlabeled dataset U i ={(u1),…,(u n )}.

[0049] In traditional semi-supervised learning methods, both labeled and unlabeled data are used for learning, and a set of model parameters are shared during training. However, this approach is not suitable for scenarios where labels are stored on the server, as it may cause the model to forget the knowledge learned from the labeled data. To address this issue, the present invention decomposes the model parameters to handle supervised learning and unsupervised learning separately. The model parameter ω is decomposed into two variables: σ for supervised learning and ψ for unsupervised learning, i.e., ω = σ + ψ.

[0050] For learning with labeled data, standard supervised learning is performed on σ by minimizing the labeled loss, while keeping ψ unchanged during training. In each round of communication, the edge server trains the model on its stored labeled dataset and updates the model σ S , and then transmit the updated model parameters to the client. The loss function of the edge server model training is defined as follows:

[0051]

[0052] where f(x j ; σ s ) represents the edge server model σ S For the model prediction of the labeled samples x j , l(.) represents the loss function of the PoseNet model for each sample.

[0053] For the learning of the unlabeled data, the unsupervised learning is performed on the model parameters ψ, while σ is fixed during the learning phase, which is performed by minimizing the unlabeled loss term. On the client side, the semi-supervised federated learning framework combines consistency regularization and pseudo-labeling method during the training process, and is improved to better adapt to the visual positioning task.

[0054] The process of the pseudo-labeling method is as follows:

[0055] Define the edge server as k, and there are multiple clients under the server, defined as i∈{1,…,N}. For the image data x input by the user, use the image data augmentation method to obtain Common image enhancement methods include image rotation, scaling, translation, etc. x and are input into the i-th client model for forward inference to obtain the predicted value f(x; ω i ) and ω i is the model parameter of the i-th client. The calculation method of the pseudo label is as follows:

[0056]

[0057] where λ represents the weight factor of the predicted value, which is an adjustable hyperparameter, and the actual effect is adjusted. For the i-th client model, the loss function of the pseudo-label training is defined as:

[0058]

[0059] where j represents the unlabeled sample, |D i | is the number of unlabeled samples on the i-th client, l(.) is the loss function of the positioning model for each sample j, and y j represents the calculated pseudo label.

[0060] The consistency regularization process is as follows:

[0061] The unlabeled data is defined as u, and the consistency regularization is to ensure that the model prediction is stable and reliable even if there is noise in the unlabeled data. In order to increase the diversity of the data set, data enhancement methods are applied to transform the image, such as image rotation, translation, etc., to obtain the perturbed form of the data The goal of consistency regularization is to make the distance value between the model prediction value f(u; ω i ) and as small as possible, where the distance measurement uses KL (Kullback-Leibler) divergence, which is a measure of the difference between two probability distributions. Based on the KL divergence, the consistency regularization loss of the i-th client is defined as:

[0062]

[0063] Where |D i | is the number of unlabeled samples on the i-th client.

[0064] In the present application, a cloud-edge-end collaborative hierarchical visual positioning system is proposed. When the scene changes and the model needs to be updated, the system undergoes two stages of model aggregation. First, joint training is performed between the client and the edge server, and in this process, the parameters of the model are aggregated at the edge server. Then, the second stage is entered, that is, the edge server uploads the model parameters to the cloud server to complete further aggregation.

[0065] The present application proposes an aggregation algorithm HierAgg for model updating and aggregation under this framework. It is assumed that there is a cloud server, K edge servers k, and some client sets under each edge server, and there are N clients in each client set. The main steps of the HierAgg algorithm are as follows: (1) After T1 local updates at each client, each edge server aggregates the model parameters from the clients. (2) Then, after every T2 edge model aggregation, the cloud server aggregates all edge server models, which means that communication with the cloud occurs after every T1xT2 local updates. The aggregation strategy between the edge and the cloud uses the FedAvg algorithm, and the model aggregation between the client and the edge will be described in detail below. The unsupervised loss of the i-th client is defined as:

[0066]

[0067]

[0068] Where L PL is the pseudo-label loss, L CR is the consistency regularization loss, They are the L2 and L1 regularization coefficients, which are used to control the degree of influence of regularization on the model. The L2 regularization term is to prevent the unsupervised model parameter ψ from being far away from the knowledge learned by the supervised model parameter σ, that is, to make the knowledge obtained by unsupervised learning close to the knowledge learned by supervised learning. The L1 regularization term is to make ψ sparse, which can reduce the communication cost of parameter transmission in federated learning. t is a dynamic coefficient that changes with each round of communication, T is the number of communication rounds for the i-th client model training, t represents the current training round, μ t The purpose of is to balance the impact of pseudo-labels on unsupervised training. The client model training process is divided into three main phases: initial, mid, and final. In the initial phase of training, the model is not fully trained, and its performance is relatively limited, resulting in low pseudo-label accuracy. In this case, over-reliance on pseudo-labels can have a negative impact on model training. Therefore, in this phase, it is recommended to reduce the proportion of pseudo-label loss in the total loss of client model training. As training progresses to the mid-stage, model performance gradually improves, and the role of pseudo-labels begins to become positive. At this stage, it is recommended to make the influence of pseudo-label loss and consistency regularization loss similar. Finally, in the final phase of training, as the client model approaches convergence, the entire training process is gradually completed.

[0069] The client trains based on the unsupervised loss function described above. After multiple rounds of local training, the client uploads the updated model parameters to the edge server for aggregation. Since relying solely on unlabeled data for training can lead to deviations in model prediction accuracy, it is necessary to integrate the knowledge gained by the model through supervised learning. The edge server's aggregation formula is as follows:

[0070]

[0071] in, refers to the model parameters after aggregation, t refers to the communication round, Refers to the i-th client based on the unlabeled dataset D i The trained model parameters, Refers to the edge server k in the labeled dataset D k The model trained on , N refers to the number of clients.

[0072] After the model completes its aggregation at the edge, the edge server retransmits the model back to each local client. The client repeats the training and aggregation process until the predetermined number of communication rounds is reached. At this point, the model parameters are uploaded to the cloud for update. After several rounds of communication and updates, the client downloads the updated model from the edge server and uses it to perform positioning tasks.

[0073] The application further provides a device equipment comprising a memory and a processor, wherein the memory is used for storing a computer program capable of running on the processor; and the processor is used for executing the steps of the visual positioning method based on semi-supervised federated learning when the computer program is run.

[0074] The application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by at least one processor to implement the steps of the visual positioning method based on semi-supervised federated learning.

[0075] The dataset used is a public outdoor scene dataset Cambridge Landmark, which is a large-scale outdoor visual positioning dataset shot around Cambridge University and consists of 4 scenes. Figure 2 As shown in the figure, PerSSFL in the figure represents the method proposed in the application, and it can be seen from the results that, after 100 rounds of communication rounds, the positioning accuracy of PerSSFL is better than that of the centralized and FedAvg algorithms, reaching about 3.2 meters, proving the effectiveness of the application.

Claims

1. A visual positioning method based on semi-supervised federated learning, characterized in that, A hierarchical architecture of cloud server-edge server-client is adopted, and aggregation is respectively performed on the edge server and the cloud server, so that the edge server and the cloud server are both updated; the specific implementation process is as follows: When the visual scene changes, the user collects the unlabeled image data using the client device; In each communication round, the client under each edge server trains based on the unlabeled data, and adopts a model parameter decomposition strategy to distinguish between supervised learning and unsupervised learning; The pseudo-label and consistency regularization loss are fused in the unsupervised loss function; the local client performs multiple rounds of training based on the loss, and uploads the updated model parameters to the edge server side; In each communication round, the edge server side comprehensively considers the influence of the labeled data and the unlabeled data, and performs model parameter aggregation according to the aggregation algorithm HierAgg; After multiple communication rounds, the updated visual positioning model is obtained; the client uses the updated model for positioning; The implementation process of the model parameter aggregation according to the aggregation algorithm HierAgg is as follows: After T1 local updates are performed on each client, each edge server aggregates the model parameters from the clients; after T2 edge model aggregations are performed, the cloud server aggregates the models of all edge servers; the aggregation strategies of the edge side and the cloud side use the FedAvg algorithm, and the aggregation formula between the edge side and the clients is as follows: wherein, denotes the model parameters after the aggregation, t denotes the communication round, denotes the model parameters trained by the i-th client based on the unlabelled dataset D i denotes the model parameters trained by the i-th client based on the unlabelled dataset D denotes the model parameters trained by the k-th edge server on the labelled dataset D k denotes the model parameters trained by the k-th edge server on the labelled dataset D 2. The visual positioning method based on semi-supervised federated learning according to claim 1, characterized in that, The implementation process of the supervised learning is as follows: The model parameters ω are decomposed into two variables, σ for supervised learning and ψ for unsupervised learning, i.e., ω=σ+ψ; For learning of the labeled data, σ is subjected to standard supervised learning by minimizing the labeled loss, and ψ remains unchanged during the training process; the loss function of the edge server side model training is defined as follows: where f(x j ; σ s ) denotes the edge server model σ S For model predictions on labeled samples x j , l(.) denotes the loss function of the PoseNet model for each sample.

3. The visual positioning method based on semi-supervised federated learning according to claim 1, characterized in that, The implementation process of the unsupervised learning is as follows: The model parameters ψ are subjected to unsupervised learning, and σ is kept fixed during the learning stage; the unsupervised loss function is improved by combining the consistency regularization and the pseudo-label method at the client side; Define the edge server as k, and there are multiple clients under the server, defined as i∈{1,…,N}; for the image data x input by the user, use the image data enhancement method to obtain x Input into the i-th client model for forward inference to obtain the predicted value f(x;ω i ) and ω i The model parameters of the i-th client; the calculation method of the pseudo label is: Wherein, λ represents a weight factor of a predicted value; for the i-th client model, the loss function of the pseudo-label training is as follows: where j refers to the unlabeled samples, |D i | is the number of unlabeled samples on the i-th client, l(.) is the loss function of the localization model for each sample j, y i refers to the calculated pseudo-label; The goal of consistency regularization is to make the distance between the model predicted value f(u; ω i ) of the sample u and the model predicted value of the perturbed sample as small as possible, where the distance is measured using the KL divergence; based on the KL divergence, the consistency regularization loss of the i-th client is: where |D i | is the number of unlabeled samples on the i-th client; The unsupervised loss of the i-th client is as follows: where L PL denotes pseudo-label loss, L CR denotes consistency regularization loss, are L2, L1 regularization term coefficients, respectively; μ t is a dynamic coefficient that changes with each communication round, T is the communication round number of the i-th client model training, t represents the current training round, μ t The role of μ is to balance the influence of pseudo-labels on unsupervised training.

4. The visual positioning method based on semi-supervised federated learning according to claim 1, characterized in that, The visual positioning model is a PoseNet model.

5. An apparatus device comprising: The computer program is stored in the memory and can be run on the processor. The processor is configured to execute the steps of the visual positioning method based on semi-supervised federated learning according to any one of claims 1 to 4 when the computer program is run. The computer program is stored in the memory and can be run on the processor.

6. A storage medium, characterized by The computer program is stored in the memory and can be run on the processor.

Images