Modular functional system of a joint learning platform

Abstract

The application discloses a modular function system of a joint learning platform and belongs to the technical field of joint learning platforms, comprising a thing connection access, a local server and a joint learning platform, wherein the thing connection access is connected with the local server through a wireless network, the local server is connected with the joint learning platform through a wireless network, the thing connection access is used for collecting and acquiring data, the local server is used for pre-processing the acquired data and storing the data in a local database, and the data is subjected to corresponding training and management.The modular function system of the joint learning platform is designed in a modular manner, on the one hand, each part can be fully decoupled while closely interacting, and on the other hand, the modular manner is convenient for optimizing and iterating the modules on the basis of the modular function system in the future, without modifying the overall basic framework, and can be flexibly applied in actual use, can provide more fair and efficient joint training and expand the ecological value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of joint learning platform, and particularly relates to a modular function system of a joint learning platform. BACKGROUND

[0002] Modular design, simply speaking, is not to input computer statements and instructions one by one at the beginning, but to describe the main structure and flow of software by using main programs, subprograms, sub-processes and other frameworks, and to define and debug the input and output link relationships between the frameworks. The result of step-by-step refinement is a series of algorithm descriptions in the form of functional blocks. The method of program design in the form of functional blocks to realize the solving algorithm is called modularization. The purpose of modularization is to reduce the complexity of the program, and to simplify the operations such as program design, debugging and maintenance. Changing a sub-function only needs to change the corresponding module.

[0003] At present, the modular function design is not applied to the joint learning platform, so that the existing learning platform cannot optimize and iterate the modules, the joint training efficiency is low, and the ecological value is not good. SUMMARY

[0004] The purpose of the present application is to provide a modular function system of a joint learning platform, which is designed in a modular way. On the one hand, each part can be fully decoupled while being closely interactive. On the other hand, the modular way also facilitates the optimization and iteration of the modules in the future without modifying the overall infrastructure. In actual use, it can be flexibly applied to provide more fair and efficient joint training and expand the ecological value to solve the problems in the background technology.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] The modular function system of the joint learning platform comprises a thing connection access, a local server and a joint learning platform, the thing connection access is connected with the local server through a wireless network, and the local server is connected with the joint learning platform through a wireless network, wherein:

[0007] The thing connection access is used for collecting and acquiring data;

[0008] The local server is used for preprocessing the acquired data and storing the data in a local database, and performing corresponding training and management on the data;

[0009] The joint learning platform is used for uniformly arranging and managing various joint training tasks and agents, and plays a central coordinating role.

[0010] Further, the Internet of Things access includes machine equipment, edge box and camera, and the machine equipment, edge box and camera all acquire data through the data standardization Internet of Things interface, and store the acquired data in the local database after data cleaning operation, and the user can operate offline without strong coupling in the joint training process.

[0011] Further, the local server includes Internet of Things system, local training configuration, local training agent and local resource management cooperation, the Internet of Things system includes visualization application, data analysis, local database, data preprocessing and data acquisition, the acquired data is stored in the local database after processing through data preprocessing, and the data of the visualization application is also stored in the local database after analysis through data analysis;

[0012] The local training configuration includes data / model / task query, joint task configuration and local model deduction, the data / model / task query is used for querying data / models / tasks, the joint task configuration is used for configuring and managing joint tasks, and the local model deduction is used for deducing local models;

[0013] The local training agent includes Agent local deployment, local model training, abnormal processing strategy, training data acquisition, uploading / downloading model and monitoring information feedback, the Agent local deployment is used for deploying Agent local data, the local model training is used for training local models, the abnormal processing strategy is used for processing strategy of abnormal conditions, the training data acquisition is used for converting the acquired data into the format required by the algorithm, the uploading / downloading model is used for uploading or downloading the model, and the monitoring information feedback is used for monitoring information and feeding back the monitored information;

[0014] The local resource management cooperation includes Agent deployment, distributed communication, model deployment, resource monitoring, resource virtualization, resource scheduling, management and control strategy, resource data and monitoring data, the Agent deployment is used for deploying Agent, the distributed communication is used for connecting the joint learning platform and transmitting data, the model deployment is used for deploying models, the resource monitoring is used for monitoring resources, the resource virtualization is used for virtualizing resources, the resource scheduling is used for scheduling resources and reasonably allocating resources, the management and control strategy is used for managing and controlling data, the resource data is used for viewing when managing and controlling data, and the monitoring data is used for monitoring data.

[0015] Further, the joint learning platform comprises an intelligent ecosystem, a joint learning plan, a joint learning engine and platform support, the intelligent ecosystem comprises a data information library, a resource information library, a model information library, a model storage library, a contribution value measurement tracking mechanism and a fair incentive mechanism, the data information library stores data information, the resource information library stores resource information, the model information library stores model information, the model storage library is used for storing models, the contribution value measurement tracking mechanism is used for measuring and tracking contribution value, and the fair incentive mechanism is used for fairly incentivizing joint learning.

[0016] The joint learning plan comprises participant selection, model integration deployment, model aggregation strategy and accuracy value feedback, the participant selection is used for selecting a joint learning module, the model integration deployment is used for integrating and managing and deploying a model, the model aggregation strategy is used for aggregating a model, and the accuracy value feedback is used for feeding back accuracy value of joint learning.

[0017] The joint learning engine comprises an ML / DL algorithm, distributed exception processing, an aggregation strategy, a privacy security protocol and an adaptive mechanism, the ML / DL algorithm is used for calculating joint learning, the distributed exception processing is used for processing exceptions of joint learning, the aggregation strategy is used for aggregating modules of joint learning, the privacy security protocol is used for ensuring privacy security of a joint learning module, and the adaptive mechanism is used for enabling a joint learning module to quickly adapt to a joint learning platform.

[0018] The platform support comprises resource network topology analysis, virtualization remote deployment, resource decision, distributed communication, resource description, SLA data and resource operation, the resource network topology analysis is used for analyzing resource network topology, the virtualization remote deployment is used for remotely deploying a joint learning module, the resource decision is used for managing resources, the distributed communication is used for connecting local resource management and data transmission, the resource description is used for describing resources, the SLA data is used for supporting a platform, and the resource operation is used for operating learning of resources.

[0019] Further, an output end of the Internet of Things access is connected with an input end of the Internet of Things system, an output end of the Internet of Things system is connected with an input end of the local training configuration, the local training configuration is connected with the joint learning plan, the joint learning plan is connected with the joint learning engine, the joint learning engine is connected with the platform support, the platform support is connected with local resource management, and the joint learning engine is further connected with the local training agent.

[0020] Further, the data preprocessing comprises the following steps:

[0021] S10: After data collection, the data is converted into identifiable data by a data conversion unit and stored in a temporary storage unit.

[0022] S20: The data detection unit detects the data stored in the temporary storage unit, and feeds back the detection result to the data execution unit;

[0023] S30: If the data detected by the data detection unit does not conform to the network security management regulations, the data execution unit controls the data cleaning unit to clean it up;

[0024] S40: If the data detected by the data detection unit conforms to the network security management regulations, the data execution unit transmits the data to the local database, and stores the data through the local database.

[0025] Further, the data in S40 is sorted in a clustering manner, and the sorted data information is compressed and fused and stored in the local database.

[0026] Further, the abnormal processing strategy includes the following steps:

[0027] S10: Store the local training data through the transceiver unit;

[0028] S20: The abnormal training data detection unit detects the training data, and feeds back the detection result to the abnormal training data decision unit;

[0029] S30: If the training data detected by the abnormal training data detection unit is abnormal, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine through the joint learning engine, and use the distributed abnormal processing to process the abnormal training data;

[0030] S40: If the training data detected by the abnormal training data detection unit is normal, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine through the joint learning engine, and use the aggregation strategy to aggregate the normal training data.

[0031] Further, the training data abnormality in S20 includes system, data abnormality or node interruption.

[0032] Further, the data preprocessing includes the following steps:

[0033] S10: After data collection, the data conversion unit converts it into identifiable data and stores it in the temporary storage unit;

[0034] S20: The data detection unit detects the data stored in the temporary storage unit, and feeds back the detection result to the data execution unit;

[0035] S30: If the data detected by the data detection unit does not conform to the network security management regulations, the data execution unit controls the data cleaning unit to clean it up;

[0036] S40: If the data detected by the data detection unit conforms to the network security management regulations, the data execution unit transmits the data to the local database, and stores the data through the local database;

[0037] S50: The storage space detection unit detects the storage space of the local database, and feeds back the detection result to the storage space execution unit;

[0038] S60: If the storage space detected by the storage space detection unit is less than the set threshold value, the storage space execution unit transmits an instruction to the local database, so that the local database receives and stores the data;

[0039] S70: If the storage space detected by the storage space detection unit is equal to the set threshold value, the storage space execution unit transmits an instruction to the storage space cleaning unit, and the storage space cleaning unit stores the received new data in the local database after deleting the stored data in the local database through the priority algorithm.

[0040] Compared with the prior art, the beneficial effects of the present application are:

[0041] The modular function system of the joint learning platform of the present application is designed in a modular manner, on the one hand, each part can be fully decoupled while closely interacting, on the other hand, the modular manner also facilitates the optimization iteration of the module on this basis in the future, without modifying the overall basic framework, which is flexible in actual use and can provide more fair and efficient joint training and expand the ecological value. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a schematic diagram of the modular function system of the joint learning platform of the present application;

[0043] Figure 2 is a flowchart of data preprocessing of embodiment one of the present application;

[0044] Figure 3 is a module block diagram of data preprocessing of embodiment one of the present application;

[0045] Figure 4 is a flowchart of the abnormal processing strategy of the present application;

[0046] Figure 5 is an algorithm diagram of the abnormal processing strategy of the present application;

[0047] Figure 6 is a flowchart of data preprocessing of embodiment two of the present application;

[0048] Figure 7 is a module block diagram of data preprocessing of embodiment two of the present application;

[0049] Figure 8 Algorithmic diagram for data preprocessing of embodiment two of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0051] Embodiment one

[0052] Referring to Figure 1 The modular function system of the joint learning platform includes the Internet of Things access, the local server and the joint learning platform. The Internet of Things access is connected with the local server through a wireless network. The local server is connected with the joint learning platform through a wireless network. The Internet of Things access is used for collecting data.

[0053] The Internet of Things access is used for collecting data.

[0054] The local server is used for preprocessing the collected data and storing the data in the local database, and training and managing the data.

[0055] The joint learning platform is used for uniformly arranging and managing various joint training tasks and agents, and plays a central coordination role.

[0056] The Internet of Things access includes machine equipment, edge boxes and cameras. The machine equipment, edge boxes and cameras all acquire data through a data standardization Internet of Things interface, and store the acquired data in the local database after data cleaning operation. Users can operate offline and do not need to be strongly coupled in the joint training process.

[0057] The local server includes an Internet of Things system, a local training configuration, a local training agent and local resource management collaboration. The Internet of Things system includes a visualization application, data analysis, a local database, data preprocessing and data acquisition. The collected data is stored in the local database after being processed by data preprocessing. The data of the visualization application is also stored in the local database after being analyzed by data analysis.

[0058] The local training configuration includes data / model / task query, joint task configuration and local model deduction. The data / model / task query is used for querying data / models / tasks. The joint task configuration is used for configuring and managing joint tasks. The local model deduction is used for deducing local models.

[0059] The local training agent includes agent local deployment, local model training, abnormal handling strategy, training data acquisition, model uploading / downloading and monitoring information feedback. The agent local deployment is used for deploying the agent local data. The local model training is used for training the local model. The abnormal handling strategy is used for the strategy for handling abnormal conditions. The training data acquisition is used for converting the acquired data into the required format of the algorithm. The model uploading / downloading is used for uploading or downloading the model. The monitoring information feedback is used for monitoring the information and feeding back the monitored information.

[0060] The local resource management cooperation includes agent deployment, distributed communication, model deployment, resource monitoring, resource virtualization, resource scheduling, management and control strategy, resource data and monitoring data. The agent deployment is used for deploying the agent. The distributed communication is used for connecting the joint learning platform and transmitting data. The model deployment is used for deploying the model. The resource monitoring is used for monitoring the resources. The resource virtualization is used for virtualizing the resources. The resource scheduling is used for scheduling the resources and reasonably allocating the resources. The management and control strategy is used for managing and controlling the data. The resource data is used for viewing when managing and controlling the data. The monitoring data is used for monitoring the data.

[0061] In the whole process of joint training, the most core is the customer side agent, which carries the whole logic of user local training, including the local deployment and start logic of the agent, training data acquisition, local model training logic, uploading and downloading model interface calling, various local abnormal handling and the like. These modules jointly support the model training and processing logic of various algorithms supported by the agent. The functions of the agent will be further optimized and strengthened on the basis of these modules. The local resource management cooperation has various cloud basic services, mainly providing local monitoring and resource scheduling and distribution, and interacting with the engine and the agent through the management console.

[0062] The joint learning platform includes an intelligent ecosystem, a joint learning plan, a joint learning engine and platform support. The intelligent ecosystem includes a data information library, a resource information library, a model information library, a model storage library, a contribution value measurement tracking mechanism and a fair incentive mechanism. The data information library stores data information. The resource information library stores resource information. The model information library stores model information. The model storage library is used for storing models. The contribution value measurement tracking mechanism is used for measuring and tracking the contribution value. The fair incentive mechanism is used for fairly incentivizing joint learning.

[0063] The joint learning plan comprises participant selection, model integrated deployment, model aggregation strategy and accuracy value feedback, the participant selection is used for selecting the joint learning module, the model integrated deployment is used for integrated management and deployment of the model, the model aggregation strategy is used for aggregating the model, and the accuracy value feedback is used for feeding back the accuracy value of the joint learning;

[0064] The joint learning engine comprises an ML / DL algorithm, distributed exception processing, an aggregation strategy, a privacy security protocol and an adaptive mechanism, the ML / DL algorithm is used for computing the joint learning, the distributed exception processing is used for processing the exception of the joint learning, the aggregation strategy is used for aggregating the module of the joint learning, the privacy security protocol is used for ensuring the privacy security of the joint learning module, and the adaptive mechanism is used for enabling the joint learning module to quickly adapt to the joint learning platform.

[0065] The platform support comprises resource network topology analysis, virtualization remote deployment, resource decision, distributed communication, resource description, SLA data and resource operation, the resource network topology analysis is used for analyzing the topology of the resource network, the virtualization remote deployment is used for remotely deploying the joint learning module, the resource decision is used for managing the resource, the distributed communication is used for connecting the local resource management and transmitting data, the resource description is used for describing the resource, the SLA data is used for supporting the platform, and the resource operation is used for operating and learning the resource.

[0066] The output end of the IOT access is connected with the input end of the IOT system, the output end of the IOT system is connected with the input end of the local training configuration, the local training configuration is connected with the joint learning plan, the joint learning plan is connected with the joint learning engine, the joint learning engine is connected with the platform support, the platform support is connected with the local resource management, and the joint learning engine is further connected with the local training agent.

[0067] The closest partner of the Agent is the central joint learning engine, which can be said to be the key core of the entire joint learning. The engine, as the central node, will be deployed in the cloud and be responsible for unified arrangement and management of various joint training tasks and Agents. The central coordination function will perform a series of aggregation operations on the models uploaded by each local node, and the engine will uniformly handle them. The engine needs to be able to incorporate joint machine learning, joint deep learning and other basic algorithms based on the same joint learning framework, including various horizontal and vertical joint modes. From the perspective of training safety, it also needs to consider the way of privacy encryption. From the perspective of computational efficiency, it needs to consider distributed processing systems and various exception handling. These are the key functional modules that the engine needs to provide to ensure the integrity of joint training and platform support services. The main interaction is between resource decision and engine, and other modules are the internal basic modules of the cloud platform. Of course, they serve not only joint learning, so modular design is also beneficial to the flexible application of the platform to some extent.

[0068] Referring to Figures 2-3 , the data preprocessing comprises the following steps:

[0069] S10: After data collection, the data is converted into identifiable data by a data conversion unit and stored in a temporary storage unit;

[0070] S20: The data stored in the temporary storage unit is detected by a data detection unit, and the detection result is fed back to a data execution unit;

[0071] S30: If the data detected by the data detection unit does not comply with the network security management regulations, the data execution unit will control the data cleaning unit to clean it up;

[0072] S40: If the data detected by the data detection unit complies with the network security management regulations, the data execution unit will transmit the data to the local database, store the data through the local database, sort the data in a clustering manner, and store the compressed and fused data information in the local database.

[0073] Referring to Figures 4-5 , the abnormal processing strategy comprises the following steps:

[0074] S10: Store the local training data through a transceiver unit;

[0075] S20: Detect the training data through an abnormal training data detection unit, and feed back the detection result to an abnormal training data decision unit. The training data anomaly in S20 includes system, data anomaly or node interruption;

[0076] S30: If the training data detected by the abnormal training data detection unit is abnormal, the abnormal training data decision unit will transmit feedback to the joint learning engine through the monitoring information, and use distributed exception handling to process the abnormal training data;

[0077] S40: If the training data detected by the abnormal training data detection unit is normal, the abnormal training data decision unit will transmit feedback to the joint learning engine through the monitoring information, and use aggregation strategy to aggregate the normal training data.

[0078] Embodiment two

[0079] Referring to Figure 1 , the modular function system of the joint learning platform includes the Internet of Things access, the local server and the joint learning platform, the Internet of Things access is connected with the local server through the wireless network, and the local server is connected with the joint learning platform through the wireless network, wherein:

[0080] The Internet of Things access is used for collecting and acquiring data;

[0081] The local server is used for pre-processing the acquired data and storing the data in the local database, and performing corresponding training and management on the data;

[0082] The joint learning platform is used for unified arrangement and management of various joint training tasks and agents, and plays a central coordination role.

[0083] The Internet of Things access includes machine devices, edge boxes and cameras, and the machine devices, edge boxes and cameras all acquire data through the data standardization Internet of Things interface, and store the acquired data in the local database after data cleaning operation, and the user can operate offline without strong coupling in the joint training process.

[0084] The local server includes the Internet of Things system, the local training configuration, the local training agent and the local resource management collaboration, the Internet of Things system includes visual application, data analysis, local database, data preprocessing and data acquisition, the collected data is stored in the local database after being processed by data preprocessing, and the data of the visual application is also stored in the local database after being analyzed by data analysis;

[0085] The local training configuration includes data / model / task query, joint task configuration and local model deduction, the data / model / task query is used for querying data / models / tasks, the joint task configuration is used for configuration management of joint tasks, and the local model deduction is used for deduction of local models;

[0086] The local training agent includes agent local deployment, local model training, abnormal handling strategy, training data acquisition, model uploading / downloading and monitoring information feedback. The agent local deployment is used for deploying the agent local data. The local model training is used for training the local model. The abnormal handling strategy is used for the strategy for handling the abnormal situation. The training data acquisition is used for converting the acquired data into the required format of the algorithm. The model uploading / downloading is used for uploading or downloading the model. The monitoring information feedback is used for monitoring the information and feeding back the monitored information.

[0087] The local resource management cooperation includes agent deployment, distributed communication, model deployment, resource monitoring, resource virtualization, resource scheduling, management and control strategy, resource data and monitoring data. The agent deployment is used for deploying the agent. The distributed communication is used for connecting the joint learning platform and transmitting data. The model deployment is used for deploying the model. The resource monitoring is used for monitoring the resources. The resource virtualization is used for virtualizing the resources. The resource scheduling is used for scheduling the resources and reasonably allocating the resources. The management and control strategy is used for managing and controlling the data. The resource data is used for viewing when managing and controlling the data. The monitoring data is used for monitoring the data.

[0088] The joint learning platform includes an intelligent ecosystem, a joint learning plan, a joint learning engine and platform support. The intelligent ecosystem includes a data information library, a resource information library, a model information library, a model storage library, a contribution value measurement tracking mechanism and a fair incentive mechanism. The data information library stores data information. The resource information library stores resource information. The model information library stores model information. The model storage library is used for storing models. The contribution value measurement tracking mechanism is used for measuring and tracking the contribution value. The fair incentive mechanism is used for fairly incentivizing the joint learning.

[0089] The joint learning plan includes participant selection, model integration deployment, model aggregation strategy and accuracy value feedback. The participant selection is used for selecting the joint learning module. The model integration deployment is used for integrating and managing the deployment of the model. The model aggregation strategy is used for aggregating the model. The accuracy value feedback is used for feeding back the accuracy value of the joint learning.

[0090] The joint learning engine includes ML / DL algorithm, distributed abnormal handling, aggregation strategy, privacy security protocol and adaptive mechanism. The ML / DL algorithm is used for calculating the joint learning. The distributed abnormal handling is used for handling the abnormal situation of the joint learning. The aggregation strategy is used for aggregating the modules of the joint learning. The privacy security protocol is used for ensuring the privacy security of the joint learning module. The adaptive mechanism is used for enabling the joint learning module to quickly adapt to the joint learning platform.

[0091] The platform support includes resource network topology analysis, virtualized remote deployment, resource decision, distributed communication, resource description, SLA data, and resource operation. The resource network topology analysis is used for topology analysis of a resource network. The virtualized remote deployment is used for remote deployment of a joint learning module. The resource decision is used for management of resources. The distributed communication is used for connection of local resource management collaboration and data transmission. The resource description is used for description of resources. The SLA data is used for support of a platform. The resource operation is used for operation learning of resources.

[0092] The output end of the Internet of Things access is connected with the input end of the Internet of Things system. The output end of the Internet of Things system is connected with the input end of the local training configuration. The local training configuration is connected with the joint learning plan. The joint learning plan is connected with the joint learning engine. The joint learning engine is connected with the platform support. The platform support is connected with the local resource management collaboration. The joint learning engine is also connected with the local training agent.

[0093] Referring to Figures 6-8 The data preprocessing includes the following steps:

[0094] S10: After data collection, the data is converted into identifiable data by a data conversion unit and stored in a temporary storage unit.

[0095] S20: The data stored in the temporary storage unit is detected by a data detection unit, and the detection result is fed back to a data execution unit.

[0096] S30: If the data detected by the data detection unit does not comply with the network security management regulations, the data execution unit will control the data cleaning unit to clean it up.

[0097] S40: If the data detected by the data detection unit complies with the network security management regulations, the data execution unit will transmit the data to the local database. The data is stored by the local database, sorted in a clustering manner, and compressed and fused after sorting of the data information, and stored in the local database.

[0098] S50: The storage space of the local database is detected by a storage space detection unit, and the detection result is fed back to a storage space execution unit.

[0099] S60: If the storage space detected by the storage space detection unit is less than the set threshold, the storage space execution unit transmits an instruction to the local database, so that the local database receives and stores the data.

[0100] S70: If the storage space detected by the storage space detection unit is equal to the set threshold, the storage space execution unit transmits an instruction to the storage space cleaning unit, and the storage space cleaning unit stores the received new data in the local database after deleting the data stored in the local database by the priority algorithm.

[0101] The abnormality processing strategy comprises the following steps:

[0102] S10: Store the local training data through the transceiving unit;

[0103] S20: Detect the training data through the abnormal training data detection unit, and feed back the detection result to the abnormal training data decision unit, wherein the training data abnormality in S20 includes system, data abnormality or node interruption;

[0104] S30: If the training data detected by the abnormal training data detection unit is abnormal, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine, and use the distributed abnormality processing to process the abnormal training data;

[0105] S40: If the training data detected by the abnormal training data detection unit is normal, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine, and use the aggregation strategy to aggregate the normal training data

[0106] To sum up, the modular function system of the joint learning platform of the present application is designed in a modular manner, on the one hand, each part can be fully decoupled, while being closely interactive, on the other hand, the modular manner also facilitates the optimization iteration of the modules on this basis in the future, without modifying the overall basic architecture, which can provide more fair and efficient joint training and expand the ecological value.

[0107] The above is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can make equivalent replacement or change according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application, which should be covered in the protection scope of the present application.

Claims

1. A modular functional system of a federated learning platform, comprising IoT access, a local server, and a federated learning platform, characterized in that: The IoT access is connected to the local server via a wireless network, and the local server is connected to the federated learning platform via a wireless network, wherein: IoT access is used to collect and acquire data; The local server is used to preprocess the acquired data and store the data in the local database, and to perform corresponding training and management on the data. The federated learning platform is used to uniformly orchestrate and manage various federated training tasks and agents, playing a central coordinating role. The local server includes an IoT system, local training configuration, local training agent, and local resource management collaboration. The IoT system includes visualization application, data analysis, local database, data preprocessing, and data acquisition. The acquired data is processed by data preprocessing and then stored in the local database, and the data from the visualization application is analyzed by data analysis and then stored in the local database. Local training configuration includes data / model / task query, joint task configuration, and local model inference. Data / model / task query is used to query data / model / tasks, joint task configuration is used to manage the configuration of joint tasks, and local model inference is used to infer local models. The local training agent includes Agent local deployment, local model training, anomaly handling strategy, training data acquisition, model upload / download, and monitoring information feedback. Agent local deployment is used to deploy Agent local data, local model training is used to train the local model, anomaly handling strategy is used to handle abnormal situations, training data acquisition is used to convert the acquired data into the format required by the algorithm, model upload / download is used to upload or download the model, and monitoring information feedback is used to monitor information and provide feedback on the monitored information. The local resource management collaboration includes Agent deployment, distributed communication, model deployment, resource monitoring, resource virtualization, resource scheduling, control strategies, resource data, and monitoring data. Agent deployment is used to deploy agents; distributed communication is used to connect to the federated learning platform and transmit data; model deployment is used to deploy models; resource monitoring is used to monitor resources; resource virtualization is used to virtualize resources; resource scheduling is used to schedule resources and allocate them reasonably; control strategies are used to control data; resource data is used to view data during control; and monitoring data is used to monitor data. The federated learning platform includes an intelligent ecosystem, a federated learning plan, a federated learning engine, and platform support. The intelligent ecosystem includes a data information repository, a resource information repository, a model information repository, a model storage repository, a contribution value measurement and tracking mechanism, and a fair incentive mechanism. The data information repository stores data information; the resource information repository stores resource information; the model information repository stores model information; the model storage repository stores models; the contribution value measurement and tracking mechanism measures and tracks contribution value; and the fair incentive mechanism provides fair incentives for federated learning. The joint learning program includes participant selection, model integration and deployment, model aggregation strategy, and accuracy value feedback. Participant selection is used to select the joint learning modules, model integration and deployment is used to integrate and manage the models, model aggregation strategy is used to aggregate the models, and accuracy value feedback is used to provide feedback on the accuracy value of the joint learning. The federated learning engine includes ML / DL algorithms, distributed anomaly handling, aggregation strategies, privacy and security protocols, and adaptive mechanisms. ML / DL algorithms are used to perform computations on federated learning, distributed anomaly handling is used to handle anomalies in federated learning, aggregation strategies are used to aggregate federated learning modules, privacy and security protocols are used to ensure the privacy and security of federated learning modules, and adaptive mechanisms are used to enable federated learning modules to quickly adapt to the federated learning platform. The platform supports resource network topology analysis, virtualized remote deployment, resource decision-making, distributed communication, resource description, SLA data, and resource operations. Resource network topology analysis is used to analyze the topology of the resource network; virtualized remote deployment is used to remotely deploy the federated learning module; resource decision-making is used to manage resources; distributed communication is used to connect local resources for collaborative management and data transmission; resource description is used to describe resources; SLA data is used to support the platform; and resource operations are used to learn how to operate resources.

2. The modular functional system of the collaborative learning platform as described in claim 1, characterized in that, The IoT access includes machines, edge boxes, and cameras. All machines, edge boxes, and cameras acquire data through standardized IoT interfaces and store the acquired data in a local database after data cleaning. Users can operate offline without strong coupling during the joint training process.

3. The modular functional system of the collaborative learning platform as described in claim 1, characterized in that, The output terminal of the IoT access is connected to the input terminal of the IoT system, the output terminal of the IoT system is connected to the input terminal of the local training configuration, the local training configuration is connected to the joint learning plan, the joint learning plan is connected to the joint learning engine, the joint learning engine is connected to the platform support, the platform support is connected to the local resource management collaboration, and the joint learning engine is also connected to the local training agent.

4. The modular functional system of the collaborative learning platform as described in claim 3, characterized in that, The data preprocessing includes the following steps: S10: After data acquisition, the data is converted into recognizable data by the data conversion unit and stored in the temporary storage unit; S20: The data detection unit detects the data stored in the temporary storage unit and feeds back the detection results to the data execution unit; S30: If the data detected by the data detection unit does not comply with the network security management regulations, the data execution unit will control the data cleaning unit to clean it up; S40: If the data detected by the data detection unit complies with the network security management regulations, the data execution unit will transmit the data to the local database and store the data in the local database.

5. The modular functional system of the collaborative learning platform as described in claim 4, characterized in that, The data in S40 is sorted according to clustering, and the sorted data information is compressed, merged, and stored in the local database.

6. The modular functional system of the collaborative learning platform as described in claim 5, characterized in that, The exception handling strategy includes the following steps: S10: Store local training data through the transceiver unit; S20: The abnormal training data detection unit detects the training data and feeds back the detection results to the abnormal training data decision unit. S30: If the abnormal training data detection unit detects anomalies in the training data, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine and use distributed anomaly processing to process the abnormal training data. S40: If the abnormal training data detection unit detects no abnormalities in the training data, the abnormal training data decision unit will transmit the monitoring information feedback to the joint learning engine and use the aggregation strategy to aggregate the training data without abnormalities.

7. The modular functional system of the collaborative learning platform as described in claim 6, characterized in that, The training data anomalies in S20 include system anomalies, data anomalies, or node interruptions.

8. The modular functional system of the collaborative learning platform as described in claim 4, characterized in that, The data preprocessing includes the following steps: S10: After data acquisition, the data is converted into recognizable data by the data conversion unit and stored in the temporary storage unit; S20: The data detection unit detects the data stored in the temporary storage unit and feeds back the detection results to the data execution unit; S30: If the data detected by the data detection unit does not comply with the network security management regulations, the data execution unit will control the data cleaning unit to clean it up; S40: If the data detected by the data detection unit complies with the network security management regulations, the data execution unit will transmit the data to the local database and store the data in the local database; S50: The storage space of the local database is detected by the storage space detection unit, and the detection result is fed back to the storage space execution unit. S60: If the storage space detected by the storage space detection unit is less than the set threshold, the storage space execution unit sends an instruction to the local database, causing the local database to receive the data and store it. S70: If the storage space detected by the storage space detection unit is equal to the set threshold, the storage space execution unit sends an instruction to the storage space cleanup unit. The storage space cleanup unit deletes the data stored in the local database according to the priority algorithm and then stores the received new data in the local database.

Images