Multi-source heterogeneous big data acquisition and processing method based on federal learning

Abstract

The invention discloses a multi-source heterogeneous big data acquisition and processing method based on federated learning, and belongs to the technical field of data management, and the method comprises the steps: building a traffic system Internet of Things equipment data acquisition module and a data transmission module, and collecting traffic system data in real time; a local data storage and processing module is arranged in the data transmission network to receive, process and store traffic system data collected in real time; training the local data to obtain a local optimization model; the trained local optimization model parameters are uploaded to a server, and the server obtains an aggregation model based on the initialized global optimization model and the received local optimization model parameters and issues the aggregation model to the traffic system Internet of Things devices; and each traffic Internet of Things device monitors and analyzes the traffic system data in real time based on the aggregation model, obtains abnormal information types in the traffic system, and formulates corresponding response strategies of systems corresponding to the Internet of Things terminals according to different abnormities.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a method, device and computer-readable storage medium for collecting and processing multi-source heterogeneous big data based on federated learning. Background Art

[0002] With the development and widespread application of computer technology, IoT devices play an important role in smart education, medical health, smart cities, smart homes, smart transportation, etc. In smart transportation, a large number of IoT devices are connected to computer networks and rely on various types of sensors and smart devices to collect various data. However, data integration, analysis, and privacy of smart transportation still face many challenges, such as inconsistent data collection types, time misalignment, poor data quality, and low data management efficiency, including but not limited to the following aspects: Poor data privacy: In transportation application scenarios, IoT devices collect personal, corporate or geographic environment data through different types of collection devices, resulting in data disclosure to the privacy of the collection objects. Large, and difficult to supervise; Poor data availability: Different systems will be composed of multiple devices and lower systems, such as driver management systems, vehicle systems, traffic systems, weather systems, etc. The data standards and storage methods of different systems are inconsistent, which makes data integration and analysis difficult, and the accuracy, completeness and consistency of the data are poor, resulting in low reliability of data analysis; on the other hand, in terms of data processing, since multimodal data generated under multi-source heterogeneous conditions participate in model training, the application of data in machine learning is limited, and malicious data may exist, resulting in a decrease in data processing performance. At the same time, the influx of massive data at the same time causes problems such as server or terminal equipment downtime due to insufficient processing capacity. Summary of the invention

[0003] In view of the defects of the prior art, the present invention proposes a multi-source heterogeneous big data collection and processing method, device and computer-readable storage medium based on federated learning, which are used to efficiently manage and monitor the collection of smart transportation data, provide powerful technical support for the monitoring, anomaly positioning and emergency response of the transportation system, and further improve the operating efficiency, privacy and security of the intelligent transportation system.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for collecting and processing multi-source heterogeneous big data based on federated learning includes the following steps:

[0006] Step S1: Building a data collection module and a data transmission module for the Internet of Things equipment of the traffic system to collect traffic system data in real time, and setting a local data storage and processing module in the data transmission network to receive, process and store the traffic system data collected in real time;

[0007] Step S2: training the local data through a local data storage and processing unit arranged in the data transmission network to obtain a local optimization model;

[0008] Step S3: uploading the trained local optimization model parameters to the central server. The central server obtains an aggregate model based on the initialized global optimization model and the received local optimization model parameters, and sends the aggregate model to each transportation system IoT device.

[0009] Step S4: Each transportation IoT device performs real-time monitoring and analysis of the transportation system data based on the aggregation model, obtains the type of abnormal information in the transportation system, and formulates corresponding response strategies for the systems corresponding to each IoT terminal for different abnormalities.

[0010] Furthermore, the step S1 is specifically as follows:

[0011] IoT equipment data collection modules are set up at key nodes of the traffic system to collect traffic system information in real time, such as driver monitoring terminal, vehicle monitoring terminal, road condition detection terminal, etc.

[0012] A data transmission module is provided to transmit real-time data collected by the IoT device data collection module;

[0013] A local data storage and processing module is set up to receive, process and store the real-time collected traffic system data.

[0014] In this embodiment, the step S2 is specifically as follows:

[0015] Real-time traffic system data is processed through machine learning algorithms (such as neural networks, etc.), and local machine learning algorithm parameters are optimized according to the processing results. Or different machine learning algorithms are used for different traffic system data types, such as congestion information, fuel consumption information, etc., and different data structure output results are obtained according to the machine learning algorithms. The local optimization model is obtained after the output results are fused.

[0016] Furthermore, the step S3 is specifically as follows:

[0017] Step S3 integrates the multi-source data models based on the federated learning algorithm. The specific steps are as follows:

[0018] According to the number of local optimization models belonging to the IoT devices in the transportation system, N local optimization models {C1, ..., CN} are set, where each local optimization model Ci can obtain the training data stored in the local data storage and processing module, where N represents the number of local optimization models, Ci represents the i-th local optimization model, ai represents the sample data number, and a represents the total amount of sample data.

[0019] In the process of setting up the federated learning algorithm, a centralized architecture is adopted for public data (such as traffic data, weather data, etc.). In the process of setting up the federated optimization model, in each round of local optimization, all N local optimization models Ci are trained in parallel to train the sample data until convergence to obtain the optimal local optimization model, and the weights of each local optimization model are set. All local optimization model data and collected multi-source heterogeneous data are transmitted to the central server to form a central sample data pool. The central server aggregates all local optimization models according to the weighted average method to obtain an aggregated model. Finally, the aggregated model is distributed to each local optimization model for public data processing.

[0020] The above process is executed repeatedly until all local optimization models converge. At the end of the training process, each local optimization model has its own personalized model, which can be fine-tuned according to its local multi-source heterogeneous data, while optimizing the aggregation model of the central server, coordinating the aggregation model and redistributing weights between local optimization models; the specific steps are:

[0021] Step S311: Set up a central server. The central server sends a model submission request to each IoT terminal according to the aggregated model structure, data type and training requirements of the federated learning algorithm. Each IoT terminal responds to the request and uploads data to the central server. Each IoT terminal also feeds back specific requirements to the central server based on the terminal's multi-source heterogeneous data characteristics.

[0022] Step S312: The central server obtains the best matching parameter in the optimization model according to the final weight information of the received data information, and defines it as the cluster head. In the actual application process, in addition to evaluating the multi-source heterogeneous data according to the data communication quality, data structure type, data integrity, etc., the cluster head quality is comprehensively evaluated, and a new cluster head is obtained according to the evaluation results;

[0023] Step S313: clustering is performed based on the cluster head. After the cluster head is selected, the clusters formed by the remaining IoT terminals are matched with the cluster head. During the matching process, the training and model upload time of the local optimization model Ci corresponding to the IoT terminal are calculated, and the similarity between multi-source heterogeneous data is obtained, and then the clusters are divided according to the calculation results.

[0024] In addition, the matching aims to maximize the weighted average of the aggregate model. The weighted average symbolizes the relationship between the federated aggregate model and the local optimization model. Its definition varies depending on whether there is only one global model or multiple versions. In the simple case of a single aggregate model, clustering depends only on mobility, so all clusters belong to one aggregate model. When multiple clusters are involved, each IoT terminal can be trained to be the preferred model, i.e., the cluster head.

[0025] Step S314: Send the aggregation model parameters to each local optimization model and train them. After the aggregation model is sent to the IoT terminal, each local optimization model is iteratively trained for a specific cycle based on local sample data before each IoT terminal sends the update to the corresponding cluster head. Then, each cluster head aggregates the received models to obtain the aggregation model of the cluster and sends the update to the central server. The central server then aggregates the global updates of each cluster to obtain the global aggregation model. The aggregation process of the federated optimization model adopts the aggregation algorithm in the existing technology and allows the continuous increase of IoT terminals, further increasing the multi-source heterogeneous data processing capabilities. At the same time, aggregation updates are performed at the central server level. Each model version can be trained in several clusters to generate different global aggregation models, thereby improving the robustness of the aggregation model and having stronger robustness.

[0026] Step S315: Update the aggregation model parameters and evaluate the aggregation results. If the global aggregation model does not converge after several cycles or does not reach the target accuracy, repeat the above steps for most IoT terminals in the global joint model on the central server, skip the acquisition of the internal aggregation model of the cluster, and avoid covering up the differences of different local optimization models in the process of acquiring the internal aggregation model of the cluster. At the same time, obtain the similarity of the data collected by each IoT terminal, divide the clusters according to the similarity results, and iterate and merge the IoT terminals with the most similar data structures to form clusters based on the maximum number of clusters in the division process until the cluster value upper limit is reached. The fixed cluster value upper limit can improve the creation of clusters under uncertainty, control the number of clusters in the cycle process, prevent falling into the local optimal value, and improve the internal similarity of the cluster, and reduce the problem of reduced internal similarity due to excessive cluster size. The aggregation model optimized by the federation algorithm is transmitted to each IoT terminal. The local optimization model in the IoT terminal evaluates the aggregation model based on the received aggregation model combined with the real-time collected traffic system data and the received public data, and sends the evaluation results to the central server. The central server performs local or global iterations according to the evaluation results. Through the above process, the data processing volume of the central server can be effectively reduced. At the same time, the impact of local problems caused by global updates on other local optimization models with better evaluation results can be reduced.

[0027] By uploading local public data to the central server, the central server's perception of multi-source heterogeneous data is enhanced, and the convergence speed and accuracy of the aggregation model optimization process are improved. In each data exchange process, one cluster is selected as the cluster head, and the remaining IoT terminals are matched with the cluster head. At the same time, clustering based on update similarity is introduced to overcome the shortcomings of a single aggregation model that converges slowly and is prone to falling into local optimality.

[0028] For data with strong privacy (such as driver information, vehicle information, etc.), since the data privacy of each local optimization model is strong, the data cannot be transmitted to each local optimization model or central server for sharing, so a distributed federated optimization model is selected. Each local optimization model adopts a point-to-point direct connection. Each local optimization model and the traffic system jointly maintain a shared vector, which contains its own version and the last optimized version of the model used in the merging process. During the initialization process, each local optimization model is initialized to zero and continuously iterated and updated. Each time a fine-tuning step occurs, the local optimization model is updated, and the versions used by each IoT terminal are synchronized. Specifically, the distributed federated optimization process includes the following steps:

[0029] Step S321. Use the data with strong privacy among the multi-source heterogeneous data collected by the local traffic system to perform parallel local optimization training on each local optimization model. The training process adopts a machine learning algorithm, such as a neural network, and iterates to obtain the optimal local optimization model;

[0030] Step S322. Randomly select an IoT terminal from the traffic system, where the corresponding local optimization model Ci starts the training process, and sends a connection request to other IoT terminals to obtain the latest local optimization model version corresponding to it to generate an optimized vnew to replace vo ld, where vnew and vo ld represent the optimized and unoptimized versions respectively.

[0031] Step S323. Update the local optimization models corresponding to all interconnected IoT terminals, and send their weights Wi and training samples ai to the randomly selected local optimization model Ci.

[0032] Step S324: merge the model subset and the current model of the randomly selected local optimization model Ci into a separate aggregate model by weighted averaging, and then return to step 321.

[0033] The above steps use local IoT terminals to build a federated learning environment, which does not rely on a central server to control the training process. Through highly dynamic updates, more robust training of IoT terminals is achieved, and its performance is similar to that of models trained on data shared between IoT terminals.

[0034] Furthermore, the step S4 is specifically as follows:

[0035] Based on the updated local optimization model, combined with real-time traffic data, it processes but is not limited to the driving environment, road conditions, route planning, etc., and at the same time identifies risk points and abnormal points in the traffic system to enhance the promotion of intelligent transportation.

[0036] To achieve the above objective, according to another aspect of an embodiment of the present invention, a multi-source heterogeneous big data acquisition and processing device based on federated learning is provided.

[0037] The multi-source heterogeneous big data acquisition and processing device based on federated learning described in the present invention includes one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method steps as described above.

[0038] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0039] The computer-readable medium of the embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method described above is implemented.

[0040] Beneficial technical effects brought by the present invention:

[0041] 1. By building multi-source heterogeneous data collection through the perception layer, it is possible to achieve comprehensive monitoring of the real-time status of the traffic system. At the same time, it can integrate data from multiple aspects such as drivers, weather, and road conditions, process data through local optimization models, evaluate and analyze traffic information, and assist intelligent transportation in exploring risk points and anomalies. The results are then fed back to traffic participants, assisting them in formulating response plans and deploying optimization strategies, thereby improving the operating efficiency and safety of the intelligent transportation system.

[0042] 2. The present invention constructs a heterogeneous data processing method based on a federated optimization model, improves the traditional federated optimization model that only obtains the model parameters of the local optimization model during its operation, adopts optimization strategies for different data types, improves data security and privacy, and effectively solves the problem that the traditional federated optimization model cannot use valid data for efficient optimization based on different optimization strategies.

[0043] 3. In the process of multi-source heterogeneous data optimization, the present invention constructs a local optimization model cluster according to the similarity between the data, processes the data in a hierarchical mode, improves the creation of clusters under uncertainty through a fixed upper limit of cluster values, controls the number of clusters in the loop process, prevents falling into local optimal values, improves the internal similarity of the cluster, and reduces the problem of reduced internal similarity due to excessive cluster size. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of the multi-source heterogeneous big data collection and processing method based on federated learning of the present invention;

[0045] Figure 2A structural block diagram of a multi-source heterogeneous big data acquisition and processing device based on federated learning provided according to one embodiment of this specification;

[0046] Figure 3 The present invention is a structural block diagram of a computer-readable medium provided according to one embodiment of the present specification. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In the subsequent description, the use of suffixes such as "module", "component" or "unit" used to represent elements is only to facilitate the description of the present invention and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0048] To achieve the above object, the present invention provides a method for collecting and processing multi-source heterogeneous big data based on federated learning, comprising the following steps:

[0049] Step S1: Building a data collection module and a data transmission module for the Internet of Things equipment of the traffic system to collect traffic system data in real time, and setting a local data storage and processing module in the data transmission network to receive, process and store the traffic system data collected in real time;

[0050] Step S2: training the local data through a local data storage and processing unit arranged in the data transmission network to obtain a local optimization model;

[0051] Step S3: uploading the trained local optimization model parameters to the central server. The central server obtains an aggregate model based on the initialized global optimization model and the received local optimization model parameters, and sends the aggregate model to each transportation system IoT device.

[0052] Step S4: Each transportation IoT device performs real-time monitoring and analysis of the transportation system data based on the aggregation model, obtains the type of abnormal information in the transportation system, and formulates corresponding response strategies for the systems corresponding to each IoT terminal for different abnormalities.

[0053] Furthermore, the step S1 is specifically as follows:

[0054] IoT equipment data collection modules are set up at key nodes of the traffic system to collect traffic system information in real time, such as driver monitoring terminal, vehicle monitoring terminal, road condition detection terminal, etc.

[0055] A data transmission module is provided to transmit real-time data collected by the IoT device data collection module;

[0056] A local data storage and processing module is set up to receive, process and store the real-time collected traffic system data.

[0057] In this embodiment, the step S2 is specifically as follows:

[0058] Real-time traffic system data is processed through machine learning algorithms (such as neural networks, etc.), and local machine learning algorithm parameters are optimized according to the processing results. Or different machine learning algorithms are used for different traffic system data types, such as congestion information, fuel consumption information, etc., and different data structure output results are obtained according to the machine learning algorithms. The local optimization model is obtained after the output results are fused.

[0059] Furthermore, the step S3 is specifically as follows:

[0060] Step S3 integrates the multi-source data models based on the federated learning algorithm. The specific steps are as follows:

[0061] According to the number of local optimization models belonging to the IoT devices in the transportation system, N local optimization models {C1, ..., CN} are set, where each local optimization model Ci can obtain the training data stored in the local data storage and processing module, where N represents the number of local optimization models, Ci represents the i-th local optimization model, ai represents the sample data number, and a represents the total amount of sample data.

[0062] In the process of setting up the federated learning algorithm, a centralized architecture is adopted for public data (such as traffic data, weather data, etc.). In the process of setting up the federated optimization model, in each round of local optimization, all N local optimization models Ci are trained in parallel to train the sample data until convergence to obtain the optimal local optimization model, and the weights of each local optimization model are set. All local optimization model data and collected multi-source heterogeneous data are transmitted to the central server to form a central sample data pool. The central server aggregates all local optimization models according to the weighted average method to obtain an aggregated model. Finally, the aggregated model is distributed to each local optimization model for public data processing.

[0063] The above process is executed repeatedly until all local optimization models converge. At the end of the training process, each local optimization model has its own personalized model, which can be fine-tuned according to its local multi-source heterogeneous data, while optimizing the aggregation model of the central server, coordinating the aggregation model and redistributing weights between local optimization models; the specific steps are:

[0064] Step S311: Set up a central server. The central server sends a model submission request to each IoT terminal according to the aggregated model structure, data type and training requirements of the federated learning algorithm. Each IoT terminal responds to the request and uploads data to the central server. Each IoT terminal also feeds back specific requirements to the central server based on the terminal's multi-source heterogeneous data characteristics.

[0065] Step S312: The central server obtains the best matching parameter in the optimization model according to the final weight information of the received data information, and defines it as the cluster head. In the actual application process, in addition to evaluating the multi-source heterogeneous data according to the data communication quality, data structure type, data integrity, etc., the cluster head quality is comprehensively evaluated, and a new cluster head is obtained according to the evaluation results;

[0066] Step S313: clustering is performed based on the cluster head. After the cluster head is selected, the clusters formed by the remaining IoT terminals are matched with the cluster head. During the matching process, the training and model upload time of the local optimization model Ci corresponding to the IoT terminal are calculated, and the similarity between multi-source heterogeneous data is obtained, and then the clusters are divided according to the calculation results.

[0067] In addition, the matching aims to maximize the weighted average of the aggregate model. The weighted average symbolizes the relationship between the federated aggregate model and the local optimization model. Its definition varies depending on whether there is only one global model or multiple versions. In the simple case of a single aggregate model, clustering depends only on mobility, so all clusters belong to one aggregate model. When multiple clusters are involved, each IoT terminal can be trained to be the preferred model, i.e., the cluster head.

[0068] Step S314: Send the aggregation model parameters to each local optimization model and train them. After the aggregation model is sent to the IoT terminal, each local optimization model is iteratively trained for a specific cycle based on local sample data before each IoT terminal sends the update to the corresponding cluster head. Then, each cluster head aggregates the received models to obtain the aggregation model of the cluster and sends the update to the central server. The central server then aggregates the global updates of each cluster to obtain the global aggregation model. The aggregation process of the federated optimization model adopts the aggregation algorithm in the existing technology and allows the continuous increase of IoT terminals, further increasing the multi-source heterogeneous data processing capabilities. At the same time, aggregation updates are performed at the central server level. Each model version can be trained in several clusters to generate different global aggregation models, thereby improving the robustness of the aggregation model and having stronger robustness.

[0069] Step S315: Update the aggregation model parameters and evaluate the aggregation results. If the global aggregation model does not converge after several cycles or does not reach the target accuracy, repeat the above steps for most IoT terminals in the global joint model on the central server, skip the acquisition of the internal aggregation model of the cluster, and avoid covering up the differences of different local optimization models in the process of acquiring the internal aggregation model of the cluster. At the same time, obtain the similarity of the data collected by each IoT terminal, divide the clusters according to the similarity results, and iterate and merge the IoT terminals with the most similar data structures to form clusters based on the maximum number of clusters in the division process until the cluster value upper limit is reached. The fixed cluster value upper limit can improve the creation of clusters under uncertainty, control the number of clusters in the cycle process, prevent falling into the local optimal value, and improve the internal similarity of the cluster, and reduce the problem of reduced internal similarity due to excessive cluster size. The aggregation model optimized by the federation algorithm is transmitted to each IoT terminal. The local optimization model in the IoT terminal evaluates the aggregation model based on the received aggregation model combined with the real-time collected traffic system data and the received public data, and sends the evaluation results to the central server. The central server performs local or global iterations according to the evaluation results. Through the above process, the data processing volume of the central server can be effectively reduced. At the same time, the impact of local problems caused by global updates on other local optimization models with better evaluation results can be reduced.

[0070] By uploading local public data to the central server, the central server's perception of multi-source heterogeneous data is enhanced, and the convergence speed and accuracy of the aggregation model optimization process are improved. In each data exchange process, one cluster is selected as the cluster head, and the remaining IoT terminals are matched with the cluster head. At the same time, clustering based on update similarity is introduced to overcome the shortcomings of a single aggregation model that converges slowly and is prone to falling into local optimality.

[0071] For data with strong privacy (such as driver information, vehicle information, etc.), since the data privacy of each local optimization model is strong, the data cannot be transmitted to each local optimization model or central server for sharing, so a distributed federated optimization model is selected. Each local optimization model adopts a point-to-point direct connection. Each local optimization model and the traffic system jointly maintain a shared vector, which contains its own version and the last optimized version of the model used in the merging process. During the initialization process, each local optimization model is initialized to zero and continuously iterated and updated. Each time a fine-tuning step occurs, the local optimization model is updated, and the versions used by each IoT terminal are synchronized. Specifically, the distributed federated optimization process includes the following steps:

[0072] Step S321. Use the data with strong privacy among the multi-source heterogeneous data collected by the local traffic system to perform parallel local optimization training on each local optimization model. The training process adopts a machine learning algorithm, such as a neural network, and iterates to obtain the optimal local optimization model;

[0073] Step S322. Randomly select an IoT terminal from the traffic system, where the corresponding local optimization model Ci starts the training process, and sends a connection request to other IoT terminals to obtain the latest local optimization model version corresponding to it to generate an optimized vnew to replace vo ld, where vnew and vo ld represent the optimized and unoptimized versions respectively.

[0074] Step S323. Update the local optimization models corresponding to all interconnected IoT terminals, and send their weights Wi and training samples ai to the randomly selected local optimization model Ci.

[0075] Step S324: merge the model subset and the current model of the randomly selected local optimization model Ci into a separate aggregate model by weighted averaging, and then return to step 321.

[0076] Step S33: Build a local optimization model by fusing public data and private data through data fusion.

[0077] The above steps use local IoT terminals to build a federated learning environment, which does not rely on a central server to control the training process. Through highly dynamic updates, more robust training of IoT terminals is achieved, and its performance is similar to that of models trained on data shared between IoT terminals.

[0078] Furthermore, the step S4 is specifically as follows:

[0079] Based on the updated local optimization model, combined with real-time traffic data, it processes but is not limited to the driving environment, road conditions, route planning, etc., and at the same time identifies risk points and abnormal points in the traffic system to enhance the promotion of intelligent transportation.

[0080] To achieve the above objective, according to another aspect of an embodiment of the present invention, a multi-source heterogeneous big data acquisition and processing device based on federated learning is provided.

[0081] The multi-source heterogeneous big data acquisition and processing device based on federated learning described in the present invention includes one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method steps as described above.

[0082] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer readable medium is provided.

[0083] The computer-readable medium of the embodiment of the present invention stores a computer program, and when the program is executed by a processor, the method described above is implemented.

[0084] The methods executed by the above-mentioned programs can refer to the various embodiments of the method of the present invention, which will not be described in detail here.

[0085] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such actual relationship or order between these entities / operations / objects; the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0086] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only illustrative, and the units described as separate components may or may not be physically separated. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the present invention. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0087] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the multi-source heterogeneous big data acquisition and processing method based on federated learning described in each embodiment of the present invention.

[0089] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0090] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for collecting and processing multi-source heterogeneous big data based on federated learning, characterized in that: The method comprises the following steps: Step S1: Building a data collection module and a data transmission module for the Internet of Things equipment of the traffic system to collect traffic system data in real time, and setting a local data storage and processing module in the data transmission network to receive, process and store the traffic system data collected in real time; Step S2: training the local data through a local data storage and processing unit arranged in the data transmission network to obtain a local optimization model; Step S3: uploading the trained local optimization model parameters to the server, the server obtains the aggregate model based on the initialized global optimization model and the received local optimization model parameters, and sends the aggregate model to each traffic system IoT device; Step S4: Each transportation IoT device performs real-time monitoring and analysis of the transportation system data based on the aggregation model, obtains the type of abnormal information in the transportation system, and formulates corresponding response strategies for the systems corresponding to each IoT terminal for different abnormalities.

2. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 1 is characterized in that: The step S1 is specifically as follows: Set up IoT device data collection modules at key nodes of the transportation system to collect transportation system information in real time; A data transmission module is provided to transmit real-time data collected by the IoT device data collection module; A local data storage and processing module is set up to receive, process and store the real-time collected traffic system data.

3. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 1 is characterized in that: The step S2 is specifically as follows: Real-time traffic system data is processed through machine learning algorithms, and local machine learning algorithm parameters are optimized based on the processing results. Or different machine learning algorithms are used for different traffic system data types, and different data structure output results are obtained according to the machine learning algorithms. The local optimization model is obtained after the output results are fused.

4. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 3 is characterized in that: The machine learning algorithm is a neural network optimization algorithm.

5. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 1 is characterized in that: The step S3 is specifically as follows: Step S311: Setting a central server, the central server sends a model submission request to each IoT terminal according to the aggregated model structure, data type and training requirements of the federated learning algorithm, each IoT terminal responds to the request and uploads data to the central server, and each IoT terminal simultaneously feeds back specific requirements to the central server according to the terminal's multi-source heterogeneous data characteristics; Step S312: The central server obtains the best matching parameter in the optimization model according to the final weight information of the received data information, and defines it as the cluster head. In the actual application process, in addition to evaluating the multi-source heterogeneous data according to the data communication quality, data structure type, data integrity, etc., the cluster head quality is comprehensively evaluated, and a new cluster head is obtained according to the evaluation results; Step S313: clustering is performed based on the cluster head. After the cluster head is selected, the clusters formed by the remaining IoT terminals are matched with the cluster head. During the matching process, the training and model upload time of the local optimization model Ci corresponding to the IoT terminal are calculated, and the similarity between multi-source heterogeneous data is obtained, and then the clusters are divided according to the calculation results. Step S314: Send the aggregation model parameters to each local optimization model and train them. After the aggregation model is sent to the IoT terminal, each local optimization model performs iterative training for a specific cycle based on local sample data before each IoT terminal sends the update to the corresponding cluster head. Then, the updated model is sent to the corresponding cluster head. Then, each cluster head aggregates the received models to obtain the aggregation model of the cluster and sends the update to the server. The central server then aggregates the global updates of each cluster to obtain the global aggregation model. Step S315: Update the aggregation model parameters and evaluate the aggregation results. If the global aggregation model does not converge after several cycles or does not reach the target accuracy, repeat the above steps on the central server for most IoT terminals in the global federated model, skipping the acquisition of the cluster internal aggregation model, thereby avoiding the masking of differences between different local optimization models during the acquisition of the cluster internal aggregation model.

6. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 5 is characterized in that: The step S315 also includes simultaneously acquiring the similarity of the data collected by each IoT terminal, dividing the clusters according to the similarity results, and iteratively merging the IoT terminals with the most similar data structures to form clusters based on maximizing the number of clusters during the division process until the upper limit of the cluster value is reached.

7. The multi-source heterogeneous big data acquisition and processing method based on federated learning according to claim 5 is characterized in that: The step S3 further comprises the following steps: Step S321. Use the data with strong privacy among the multi-source heterogeneous data collected by the local traffic system to perform parallel local optimization training on each local optimization model. The training process adopts a machine learning algorithm and iterates repeatedly to obtain the optimal local optimization model. Step S322. Randomly select an IoT terminal from the traffic system, where the corresponding local optimization model Ci starts the training process, and sends a connection request to other IoT terminals to obtain the latest local optimization model version corresponding to it to generate an optimized vnew to replace vold, where vnew and vold represent the optimized and unoptimized versions respectively; Step S323. Update the local optimization models corresponding to all interconnected IoT terminals, and send their weights Wi and training samples ai to the randomly selected local optimization model Ci; Step S324. Merge the model subset and the current model of the randomly selected local optimization model Ci into a separate aggregate model model by weighted averaging, and then return to step S321.

8. The method for collecting and processing multi-source heterogeneous big data based on federated learning according to claim 7 is characterized in that: The step S3 also includes Step S33: Build a local optimization model by fusing public data and private data through data fusion.

9. A multi-source heterogeneous big data acquisition and processing device based on federated learning, characterized in that: The device includes one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method steps described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The stored computer program enables the electronic device to execute the steps of the multi-source heterogeneous big data acquisition and processing method based on federated learning as described in any one of claims 1 to 8.

Images