Multi-modal highway congestion identification method and system based on federated learning

Abstract

The present application relates to a kind of multi-modal highway congestion identification method and system, the method includes: A, local client carries out pre-processing to the multi-modal data of highway collected;B, based on multi-modal data, local client constructs training model to various modal data;For each training model, local client and trusted central server utilize federated learning and homomorphic encryption technology to carry out collaborative iterative training model, and output model is generated after training is completed;C, the back-end fusion of multi-modal data to output model, i.e. the decision of the classifier output trained to different modal data is fused, and the average congestion degree of model output is obtained;D, based on average congestion degree, the congestion calculation interval of model is optimized and updated, and the adaptive correction of the algorithm parameter of highway network congestion identification is realized.The method and system can improve the speed and accuracy of highway congestion identification, and ensure the security of data sharing.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of high-speed intelligent road networks, and particularly relates to a multi-modal highway congestion identification method and system based on federated learning and deep learning. BACKGROUND

[0002] With the rapid development of urbanization and economy, highways have become an indispensable part of people's life and economic development. However, with the increasing number of vehicles, the problem of highway congestion is becoming increasingly prominent, leading to low traffic efficiency, serious energy waste, and environmental pollution. Therefore, research on highway congestion identification technology is of great significance for optimizing traffic management and improving traffic efficiency. Highways are important traffic trunks with a large number of vehicles, and the problem of traffic congestion is becoming increasingly prominent. How to accurately identify the congestion of highways in a timely manner is of great significance for ensuring road traffic safety and improving road operation efficiency.

[0003] Currently, the artificial intelligence model for highway congestion identification is mainly based on a small amount of data owned by high-speed areas for training. The training data is often insufficient to obtain a high-accuracy model. In addition, considering various interests and privacy data protection factors, each high-speed data holding entity also has difficulty in aggregating cross-regional data. Traditional highway congestion identification has difficulty in ensuring the safe sharing of cross-regional data when building models in the early stage, and various privacy data in the field of highway networks cannot be guaranteed in terms of security. Once the privacy data is leaked, it will have an impact on people or regions. The above reasons all illustrate the difficulty of data utilization in the field of power Internet of Things, so it is necessary to introduce federated transfer learning to protect the safe sharing of data while solving the problem of data quantity. In addition, existing highway congestion identification methods usually analyze based on single sensor data or single modal data, which is difficult to meet the complex and variable actual traffic conditions, so a multi-modal highway congestion identification technology is needed. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal highway congestion identification method and system that can improve the speed and accuracy of highway congestion identification and ensure the security of data sharing.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows: a multi-modal highway congestion identification method, comprising the following steps:

[0006] Step A, the local client C={C i , i=1, 2,..., n} preprocesses the collected multi-modal highway data to obtain n client preprocessed data D={D i, i = 1, 2, …, n}, wherein C is a set of all highway network regional clients independently holding data, and D is a set of all data held by the clients;

[0007] Step B, based on the preprocessed multi-modal data, the local client constructs a training model for each kind of modal data; for each independent training model, the n local clients and the trusted central server perform collaborative training, iteratively train the model using federated learning and homomorphic encryption technology to realize secure sharing of highway network private data, and generate an output model M = {M z , z = 1, 2, …, q} after training, wherein M z is the model trained for the zth modal data;

[0008] Step C, based on the output model obtained in step B, performing backend fusion of multi-modal data, i.e., decision-level fusion of the output of the classifiers trained for different modal data, to obtain the average congestion degree J i of the model output;

[0009] Step D, based on the average congestion degree J i obtained in step C, optimizing and updating the congestion calculation interval of the model to realize adaptive correction of the algorithm parameters of the highway network congestion recognition.

[0010] Further, the step A specifically includes the following steps:

[0011] Step A1, cleaning and classifying the multi-modal data including images and texts collected by the local client C i ; for text data, removing duplicate values, null values and abnormal values; after the preprocessing step, obtaining the data D i corresponding to the local client;

[0012] Step A2, based on the data processed in step A1, for each highway in the congestion recognition area, for the divided decision interval [T k-1 , T k ), there are camera shooting road condition media cut frame image data or road network gantry text data in the decision interval, wherein k is the current decision time point, i represents the ith client, and l is the total number of data.

[0013] Further, the step B specifically includes the following steps:

[0014] Step B1, for the training of one kind of modal data, a trusted local client CServer is constructed;

[0015] Step B2, for this kind of modal data, the corresponding deep learning model Model is initialized;

[0016] Step B3, the local client encrypts the intermediate data after a round of training;

[0017] Step B4, the central server receives the encrypted data from the local client, performs aggregation and update operations, and then issues new parameters to the local client for the next round of training;

[0018] Step B5, the independent local client obtains the new round of training parameters issued by the central server, and decrypts the real data using the private key;

[0019] Step B6, the central server leads the collaborative training, continuously iteratively updates the local model parameters, and completes the model training based on federated learning until the termination condition is reached.

[0020] Further, when the accuracy error between the model obtained using federated learning and the model obtained without using federated learning is less than a small positive number, it is considered that the model meets the expectation and can be used for subsequent steps, otherwise retraining is performed:

[0021] E = |ACC FL -ACC ALL |<ε

[0022] Where ε is a small positive number set, ACC FL and ACC ALL are the accuracy of the model obtained using federated learning and without using federated learning, respectively, and E is the absolute value of the difference between the accuracy of the two models; thus, for the continuous iteration optimization target, the main focus is to reduce the loss of accuracy after introducing the security strategy, and the objective function is:

[0023] min E, (E < ε).

[0024] Further, in step B1, the central server initializes the configuration parameters P and issues them to the local client;

[0025] In step B2, the local client adds the configuration parameters P to the respective model to complete the initialization operation of the local model;

[0026] In step B3, the encryption method of the local client for the intermediate data is:

[0027]

[0028] Where t is the number of rounds of training, is the encrypted parameter of the i-th local client, is the intermediate data generated by the i-th client in this round, PK is the generated public key, and E(·) is the encryption function;

[0029] In step B4, the central server performs parameter updating in the following manner:

[0030]

[0031] wherein, is the parameter of the next round, t is the round of training, p i is the number of samples on the i-th client, p all is the total number of samples on the participating clients in this round, n is the total number of clients; in several rounds of training, 1 to n clients are selected to participate in each round, and the central server aggregates the collected data in a weighted average manner;

[0032] In step B5, the local client uses the private key to decrypt to obtain the real data in the following manner:

[0033]

[0034] wherein, is the latest round of parameters obtained from the central server after decryption, D(·) is a decryption function, and SK is a private key. The local client updates the parameters in the local model according to the obtained data in this round.

[0035] Further, in step C, based on the classifiers M = {M z , z = 1, 2,..., q} trained on the multi-modal data respectively in step B, the interval data in step A is input into each model to obtain the output result of the model:

[0036]

[0037] wherein, getJ() is a calculation of the predicted value of the model, is the predicted value of the model M z output at decision moment k;

[0038] Then, the predicted value output by the model M z is fused at the decision level to obtain the average congestion degree J i output by the model in the following manner:

[0039]

[0040] wherein, J i is the average congestion degree after fusion of the i-th local client model, w z is the weight of the z-th model, q is the total number of models, is the congestion degree predicted value output by the z-th model.

[0041] Further, the step D specifically includes the following steps: ​

[0042] Step D1, based on the average congestion degree J obtained in step C, the jth highway corresponding to the ith region at the decision time T k The average congestion degree is defined as corresponding to the decision interval [T k-1 , k ) whose optimization reference value is The objective function is:

[0043]

[0044] wherein, is the additional influence factor at the current decision time, is the optimization reference value at the decision time;

[0045] Step D2, based on the objective function in step D1, the additional influence factor and the optimization reference value at the current decision time are defined:

[0046]

[0047]

[0048]

[0049] wherein, c is the number of additional influence factors, and δ is a controllable optimization parameter.

[0050] The application also provides a multi-modal highway congestion identification system, comprising a local client, a network and a central server, wherein the local client and the central server each comprise a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the method steps of any one of claims 1-7 can be realized.

[0051] Further, the multi-modal highway congestion identification system comprises:

[0052] A highway network multi-modal data preparation module is configured to prepare multi-modal highway network data required for training a model, comprising a data acquisition sub-module and a data preprocessing sub-module; the data acquisition sub-module is configured to collect highway network multi-modal private data uniformly in a local region, such as monitoring video data and highway gate data; the data preprocessing sub-module is configured to filter, classify and clean the data to generate standard data that can be used for training;

[0053] A high-security collaborative training module is used for secure sharing of multiple independent data and distributed training of models, including a local model training submodule and a collaborative sharing submodule; the local model training submodule receives parameters issued from a central server and independently performs local training; the collaborative sharing submodule securely shares independent intermediate data generated by the local model using homomorphic encryption, and the central collaborative server aggregates and issues new round of training parameters; the use of homomorphic encryption enables the central server to aggregate without decryption;

[0054] A comprehensive decision optimization module is used for decision-level fusion and output optimization, including a decision-level fusion submodule and an output optimization submodule; the decision-level fusion submodule uses the model trained by the multi-modal data to perform prediction analysis, and then weightedly fuses the average congestion degree; the output optimization submodule analyzes the traffic situation in a variable interval, calculates the output threshold by an elastic variable algorithm, optimizes the calculation algorithm of the congestion degree, and obtains an output more consistent with the actual traffic condition.

[0055] Further, the highway network multi-modal data preparation module also cleans and classifies the collected image and text multi-modal data, and then removes duplicate values, null values and abnormal values from specific data; for the highway in a specific area, for the divided decision interval [T k-1 ,T k ), there are frame image data or road network gantry text data of the road condition captured by the highway camera in the decision interval;

[0056] The high-security collaborative training module performs collaborative training based on the preprocessed multi-modal data, iteratively trains the model using federated learning and homomorphic encryption technology, securely shares the data, and generates an output model M after completing the distributed training.

[0057] The comprehensive decision optimization module performs decision-level fusion and output optimization of multi-modal data based on the output model M generated by the high-security collaborative training module and the interval optimization algorithm, and generates a congestion degree output.

[0058] Compared with the prior art, the present application has the following beneficial effects: a multi-modal highway congestion identification method and system based on federated learning and deep learning are provided, which fully utilizes the highway network privacy data of multiple independent areas, overcomes the problem of small amount of special field data by combining multi-modal and federated learning, and effectively shortens the model training time by using transfer learning, thereby reducing the cost overhead. Compared with the commonly used single modal and single area training model of the prior art highway congestion identification, the federated learning technology combined with homomorphic encryption is used in the present application, which solves the cross-domain data sharing problem while ensuring the safety of the power internet of things privacy data. The data of all areas are used in model training, so that the model is more comprehensive and can help top-level decision-making to some extent. The system of the present application can be used for highway network privacy data preprocessing, cross-domain collaborative training and data sharing, comprehensive decision-making and output optimization, ensures the safe sharing of data of independent areas, and provides decision-making convenience for professionals. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the application scenario schematic diagram of the method and system of the embodiment of the present application.

[0060] Figure 2 is the system structure schematic diagram of the embodiment of the present application.

[0061] Figure 3 is the model training flowchart of the embodiment of the present application.

[0062] Figure 4 is the method implementation flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0063] The present application will be further described below in combination with the drawings and embodiments.

[0064] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0065] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or combinations thereof.

[0066] The multi-modal highway congestion identification method and system based on federated learning and deep learning provided by the application is based on a server, that is, a central server of the application.

[0067] As Figure 1 shown, the system architecture can include a plurality of local clients, a network and a server. The network is a medium to provide a communication link between the local clients and the server. The network can include various connection types, such as wired, wireless communication links or fiber optic cables, etc.

[0068] It should be understood that Figure 1 the number of local clients, networks and servers in the above is only illustrative. Depending on the implementation needs, there can be any number of local clients, networks and servers. For example, the server can be a server cluster composed of multiple servers, etc. The local client can be used to interact with the server through the network to receive or send messages, etc.

[0069] Figure 2 A structural schematic diagram of a multi-modal highway congestion identification system based on federated learning and deep learning suitable for implementing the embodiments of the present disclosure is shown.

[0070] The server involved in the multi-modal highway congestion identification system based on federated learning and deep learning can include a central processing unit, which can perform various appropriate actions and processes according to programs stored in a read-only memory or loaded from a storage part into a random access memory (RAM). In the RAM, various programs and data required for system operation are also stored. The CPU, ROM and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0071] The server can also involve a communication part including a keyboard, a mouse, a liquid crystal display, a speaker and a network interface card including a LAN (Local Area Network) card, a modem, etc.

[0072] A mobile terminal implementing various embodiments of the present application will now be described with reference to the accompanying drawings. In the following description, the suffix "module" or "part" or "unit" used for an element is merely intended for facilitating a description of the present application, and does not have by itself the meaning or function of distinguishing the element. Therefore, "module" and "part" can be used interchangeably.

[0073] The system includes a highway network multi-modal data preparation module, a high-security collaborative training module and a comprehensive decision optimization module.

[0074] The high-speed network multi-modal data preparation module is configured to prepare multi-modal high-speed network data required for training a model, and includes a data collection submodule and a data preprocessing submodule. The data collection submodule collects high-speed network multi-modal private data in a local region, such as monitoring video data and high-speed gate data. The data preprocessing submodule filters, classifies, and cleanses the data to generate standard data that can be used for training.

[0075] The high-security collaborative training module is configured to securely share multiple independent data and perform distributed training of a model, and includes a local model training submodule and a collaborative sharing submodule. The local model training submodule receives parameters issued from a central server and independently performs local training. The collaborative sharing submodule securely shares independent intermediate data generated by the local model using homomorphic encryption, and the central collaborative server aggregates and issues new training parameters. The use of homomorphic encryption enables the central collaborative server to perform aggregation operations without decryption.

[0076] The comprehensive decision optimization module is configured to perform fusion and output optimization at the decision level, and includes a decision-level fusion submodule and an output optimization submodule. The decision-level fusion submodule uses the model trained from multi-modal data to perform prediction analysis and then performs weighted fusion to obtain an average congestion level. The output optimization submodule analyzes traffic conditions in a variable interval, calculates an output threshold value using an elastic variable algorithm, optimizes the congestion level calculation algorithm, and obtains an output that is more consistent with actual traffic conditions.

[0077] The following will give a possible embodiment to its specific implementation scheme non-limiting description.

[0078] The high-speed network multi-modal data preparation module also cleanses and classifies the collected image and text multi-modal data, and then removes duplicate values, null values, and abnormal values from specific data. For a specific region of a highway, for a divided decision interval [T k-1 ,T k ), there are road condition media cut frame image data or road network gate text data captured by a camera in the decision interval.

[0079] The high-security collaborative training module performs collaborative training based on the preprocessed multi-modal data, iteratively trains a model using federated learning and homomorphic encryption technology, securely shares data, and generates an output model M after completing distributed training.

[0080] The comprehensive decision optimization module performs decision-level fusion and output optimization of multi-modal data based on the output model M generated by the high-security collaborative training module and an interval optimization algorithm, and generates a congestion level output.

[0081] The application makes full use of the highway network privacy data of multiple independent areas, overcomes the problem of small amount of special field data to a certain extent by means of multi-modal, federated learning and the like, and effectively shortens the model training time by using transfer learning, thereby reducing the cost overhead.

[0082] Compared with the single-modal and single-area training model commonly used in the prior art, the federated learning technology combined with homomorphic encryption is used to solve the cross-domain data sharing problem and ensure the safety of the power Internet of Things privacy data. The data of all areas are comprehensively used in model training, so that the model is more comprehensive and provides decision convenience for professionals.

[0083] The embodiment also provides a multi-modal highway congestion identification method based on the multi-modal highway congestion identification system, as shown in Figure 3 and Figure 4 , comprising the following steps:

[0084] Step A, the local client C={C i ,i=1,2,…,n} preprocesses the collected multi-modal highway data, and n pieces of preprocessed data D={D i ,i=1,2,…,n} of the client can be obtained, wherein C is a set of all independent highway network area clients holding data, and D is a set of all client-held data.

[0085] Step B, based on the preprocessed multi-modal data, the local client constructs a training model for each kind of modal data; for each independent training model, the n local clients and the trusted central server cooperatively train the model, iteratively train the model by using federated learning and homomorphic encryption technology to realize safe sharing of the highway network privacy data, and generate an output model M={M z ,z=1,2,…,q} after training, wherein M z is a model trained by the zth kind of modal data.

[0086] Step C, based on the output model obtained in step B, the back-end fusion of the multi-modal data is performed, that is, the decision-level fusion of the outputs of the classifiers trained for different modal data is performed, and the average congestion degree J i of the model output is obtained.

[0087] Step D, based on the average congestion degree J i obtained in step C, the congestion calculation interval of the model is optimized and updated, and the adaptive correction of the algorithm parameters of the highway network congestion identification is realized.

[0088] In the embodiment, the step A specifically comprises the following steps:

[0089] Step A1, collect the data of local client C i The collected multi-modal data such as images, texts, etc. are cleaned and classified. In particular, for text data, duplicate values, null values and outliers are removed. After the preprocessing step, the data D corresponding to the local client is obtained i .

[0090] Step A2, based on the data processed in step A1, for each expressway in the congestion identification area, for the divided decision interval [T k-1 ,T k ), there are The road condition media cut-frame image data or road network gantry text data shot by the expressway camera in the decision interval, where k is the time point currently being decided (traffic congestion identification), i represents the ith client, and l is the total number of data.

[0091] In this embodiment, step B specifically includes the following steps:

[0092] Step B1, for the training of a modal data, a trusted local client CServer is constructed.

[0093] Step B2, for this modal data, the corresponding deep learning model Model is initialized.

[0094] Step B3, the local client encrypts the intermediate data after a round of training.

[0095] Step B4, after the central server receives the encrypted data from the local client, it performs aggregation and update operations, and then distributes the new parameters to the local client for the next round of training.

[0096] Step B5, the independent local client obtains the new round of training parameters distributed by the central server, and obtains the real data after decryption using the private key.

[0097] Step B6, the central server leads the collaborative training, continuously iteratively updates the local model parameters, and completes the model training based on federated learning until the termination condition is reached.

[0098] When the accuracy error between the model obtained using federated learning and the model obtained without using federated learning is less than a small positive number, it is considered that the model meets the expectations and can be used in subsequent steps, otherwise retraining is performed:

[0099] E = |ACC FL -ACC ALL | < ε

[0100] Where ε is a small positive number set, ACC FL and ACC ALLThe accuracy of the model obtained using federal learning and not using federal learning, respectively, E is the absolute value of the difference between the accuracy of the two models; thus, for the optimization objective of continuous iteration, the main focus is to reduce the loss of accuracy after introducing the security strategy, and the objective function is:

[0101] minE,(E<ε)

[0102] In step B1, the central server initializes the configuration parameter P and issues it to the local client.

[0103] In step B2, the local client adds the configuration parameter P in step B1 to the respective model to complete the initialization operation of the local model.

[0104] In step B3, the encryption method of the intermediate data by the local client is:

[0105]

[0106] Where t is the number of rounds of training, is the encrypted parameter of the i-th local client, is the intermediate data generated by the i-th client in this round, PK is the generated public key, and E(·) is the encryption function.

[0107] In step B4, the parameter updating method performed by the central server is:

[0108]

[0109] Where, is the parameter of the next round, t is the number of rounds of training, and p i is the number of samples on the i-th client, p all is the total number of samples on the participating clients in this round, and n is the total number of clients; in several rounds of training, 1 to n clients are selected to participate in each round, and the central server aggregates the collected data using weighted averaging.

[0110] In step B5, the local client uses the private key to decrypt to obtain the real data in the following manner:

[0111]

[0112] Where, is the decrypted latest round of parameters obtained from the central server, D(·) is the decryption function, SK is the private key, and the local client updates the parameters in the local model according to the data obtained in this round.

[0113] In step C, based on the classifiers M = {M zz = 1, 2, …, q}, interval data in step A is fused Input each model to obtain the output result of the model:

[0114]

[0115] wherein getJ() is a calculation model of the predicted value, is the model M z output prediction value.

[0116] Then the model M z output prediction value is fused at the decision level to obtain the average congestion degree J i output by the model. The method is:

[0117]

[0118] wherein J i is the average congestion degree after fusion of the i-th local client model, w z is the weight of the z-th model, and q is the total number of models, is the congestion degree prediction value output by the z-th model.

[0119] In this embodiment, step D specifically comprises the following steps:

[0120] Step D1, based on the average congestion degree J obtained in step C, the j-th expressway corresponding to the i-th region has an average congestion degree at the decision time T k is defined as corresponding to the decision interval [T k-1 , T k ), the optimization reference value is The objective function is:

[0121]

[0122] wherein, is an additional influence factor at the current decision time, is the optimization reference value at the decision time.

[0123] Step D2, based on the objective function in step D1, the additional influence factor and the optimization reference value at the current decision time are defined:

[0124]

[0125]

[0126]

[0127] Wherein c is the number of additional influence factors, and delta is a controllable optimization parameter.

[0128] Based on the above method, the application uses federated learning and homomorphic encryption technology to realize the safe sharing of data, so that multiple parties can realize collaborative training of the model while ensuring that their regional data is not leaked, and then the model is used for congestion identification of the highway network, and an optimization algorithm is used to give comprehensive decision suggestions.

[0129] The application makes full use of the privacy data of the highway network in multiple independent regions, and uses multi-modal and federated learning to overcome the problem of small amount of data in special fields to some extent, and can use transfer learning to effectively shorten the model training time and reduce the cost.

[0130] Compared with the single modal and single region training model commonly used in the prior art for highway congestion identification, the application uses federated learning technology combined with homomorphic encryption to solve the cross-domain data sharing problem while ensuring the security of the power internet of things privacy data. The data of each region is used in model training, so that the model is more comprehensive and can help top-level decision-making to some extent.

[0131] The application provides a decision-making framework and an optimized algorithm after model training, generates a specified number of decision suggestions based on the model output, and provides suggestion credibility and analysis based on the probability value of the model output, and optimizes the output to better meet the actual road conditions.

[0132] The system can be used for preprocessing of highway network privacy data, cross-domain collaborative training and data sharing, comprehensive decision-making and output optimization, and ensures safe sharing of data in independent regions, and provides decision-making convenience for professionals.

[0133] Based on the above-provided multi-modal highway congestion identification method and system based on federated learning and deep learning, those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0134] The block diagrams of the multi-modal highway congestion identification method and system based on federated learning and deep learning provided by the present application shown in the accompanying drawings are only functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0135] The server involved in the multi-modal highway congestion identification method and system based on federated learning and deep learning provided by the present application is the unit and algorithm steps of each example described in combination with the embodiments disclosed herein, which can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been described in the above description in general terms in terms of function. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0136] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0137] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems) and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0138] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams.Figure 1 one or more processes and / or functions described in the flow Figure 1 one or more blocks or multiple blocks.

[0139] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide steps for implementing the function described in the flow Figure 1 one or more processes and / or functions described in the flow Figure 1 one or more blocks or multiple blocks.

[0140] The above description is only the preferred embodiment of the present application, not other forms of the present application, any skilled in the art can use the above disclosed technical content to change or modify as equivalent embodiments of equivalent changes. But any simple modification, equivalent change and modification of the above embodiments according to the technical essence of the present application without departing from the technical scheme of the present application still belongs to the protection scope of the technical scheme of the present application.

Claims

1. A multimodal highway congestion identification method, characterized in that, Includes the following steps: Step A, Local client C = {C i The collected multimodal data of highways, i=1, 2, …, n}, are preprocessed to obtain preprocessed data D = {D_i, i=1, 2, …, n} from n clients. i , i=1, 2, …, n}, where C is the set of all independent data-holding highway network area clients, and D is the set of all client-held data; Step B: Based on the preprocessed multimodal data, the local client constructs a training model for each modality. For each independent training model, n local clients and a trusted central server collaboratively train the model, using federated learning and homomorphic encryption techniques to iteratively train the model to achieve secure sharing of highway network privacy data. After training, an output model is generated, M = {M z Let M = { , z = 1, 2, …, q}, where M = 1, 2, …, q}. z The model trained for the z-th modality data; Step C: Based on the output model obtained in Step B, perform backend fusion of multimodal data, that is, perform decision-level fusion of the outputs of classifiers trained separately for different modalities to obtain the average congestion level J of the model output. i ; Step D, the average congestion level J obtained from step C. i The congestion calculation range of the model is optimized and updated to achieve adaptive correction of the algorithm parameters for highway network congestion identification. Step A specifically includes the following steps: Step A1, for local client C i The collected multimodal data, including images and text, is cleaned and classified; for text data, duplicate values, null values, and outliers are removed; after completing the preprocessing steps, the corresponding data D for the local client is obtained. i ; Step A2: Based on the data processed in Step A1, for each highway in the congestion identification area, for the defined decision interval [T]... k-1 ,T k ), has Q i k = {Q iv k , v=1, 2, …, l} represents the road condition media frame image data or road network gantry text data captured by highway cameras within the decision interval, where k is the current decision-making time point, i represents the i-th client, and l is the total number of data; Step B specifically includes the following steps: Step B1: For training a type of modality of data, construct a trusted local client CServer; Step B2: For this type of modal data, initialize the corresponding deep learning model. Step B3: The local client encrypts the intermediate data after one round of training. Step B4: After receiving the encrypted data from the local client, the central server performs aggregation and update operations, and then sends the new parameters to the local client for the next round of training. Step B5: The independent local client obtains the new round of training parameters issued by the central server, and decrypts them using the private key to obtain the real data; Step B6: The central server leads the collaborative training, continuously iterating and updating the local model parameters until the termination condition is met, thus completing the federated learning-based model training. In step B1, the central server initializes the configuration parameter P and sends it to the local client; In step B2, the local client adds the configuration parameter P to its respective model to complete the initialization operation of the local model; In step B3, the local client encrypts the intermediate data using the following method: Where t is the number of training rounds, Z i t C is the encrypted parameter for the i-th local client. i t Let PK be the intermediate data generated by the i-th client in this round, and E(·) be the generated public key. In step B4, the parameter update method performed by the central server is as follows: Among them, Z i t+1 Here are the parameters for the next round, where t is the training round number and p is the training parameter. i Let p be the number of samples on the i-th client. all The total number of samples from participating clients in this round is n, where n is the total number of clients. In several rounds of training, 1 to n clients are selected to participate in each round, and the central server aggregates the collected data using a weighted average. In step B5, the local client obtains the actual data by decrypting it using the private key as follows: Among them, V i t+1 The latest round of parameters is obtained from the central server after decryption. D(·) is the decryption function, SK is the private key, and the local client updates the parameters in the local model based on the data obtained in this round. In step C, the classifier M = {M}, which was trained on the multimodal data in step B, is used... z Given the interval data Q in step A, where z = 1, 2, …, q, ... i k = {Q iv k Input each model with the following parameters: v=1, 2, …, l, and obtain the model's output. Where getJ() is the calculated predicted value of the model, J i,z k For model M at decision time k z The output predicted value; Then, model M... z The predicted output values ​​are fused at the decision level to obtain the average congestion level J of the model output. i The method is as follows: Among them, J i Let w be the average congestion level after merging the i-th local client model. z Let J be the weight of the z-th model, q be the total number of models, and J be the weight of the z-th model. i,z k This represents the predicted congestion level output by the z-th model. Step D specifically includes the following steps: Step D1: Based on the average congestion level J obtained in step C, corresponding to the j-th highway in the i-th region, at decision time T... k The average level of congestion is defined as J. ij k The corresponding decision interval [T] k-1 ,T k Its optimization reference value is P. ij k The objective function is: Among them, F ij k As an additional influencing factor at this decision point, P ij k Optimized reference values ​​for decision-making moments; Step D2: Based on the objective function in Step D1, define the additional influencing factors and the optimal reference values ​​for this decision time: Where c is the number of additional influencing factors, and δ is the controllable optimization parameter.

2. The multimodal highway congestion identification method according to claim 1, characterized in that, If the accuracy error between the model obtained using federated learning and the model obtained without federated learning is less than a very small positive number, the model is considered to meet expectations and can be used for subsequent steps; otherwise, retraining is required. Where ε is a set minimum positive number, ACC FL and ACC ALL Let E represent the accuracy of the model obtained using federated learning and without federated learning, respectively, and E be the absolute value of the difference in accuracy between the two models. Therefore, for the continuous iterative optimization objective, the main focus is on reducing the accuracy loss after introducing a security strategy, and its objective function is: 。 3. A multimodal highway congestion identification system, characterized in that, It includes a local client, a network, and a central server. The local client and the central server each include a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the method as described in claim 1 or 2.

4. The multimodal highway congestion identification system according to claim 3, characterized in that, include: The highway network multimodal data preparation module is used to prepare the multimodal highway network data required for training the model, including a data acquisition submodule and a data preprocessing submodule. The data acquisition submodule collects multimodal privacy data of the highway network in a unified manner within the local domain; The data preprocessing submodule filters, classifies, and cleans the data to generate standard data that can be used for training. The high-security collaborative training module is used for secure sharing of multiple independent data and distributed training of models. It includes a local model training submodule and a collaborative sharing submodule. The local model training submodule receives parameters from the central server and performs local training independently. The collaborative sharing submodule securely shares independent intermediate data generated by the local model using homomorphic encryption. The central collaborative server then aggregates and distributes the parameters for the next round of training. The use of homomorphic encryption allows the central writing server to perform aggregation operations without decryption. The comprehensive decision optimization module is used for decision-level fusion and output optimization, including a decision-level fusion submodule and an output optimization submodule; The decision-level fusion submodule uses a model trained on multimodal data for predictive analysis, and then weights and fuses the data to obtain the average congestion level. The output optimization submodule analyzes the traffic flow within a variable range, calculates the output threshold using a flexible and variable algorithm, optimizes the congestion calculation algorithm, and obtains an output that better reflects the actual traffic conditions.

5. The multimodal highway congestion identification system according to claim 4, characterized in that, The highway network multimodal data preparation module also cleans and classifies the collected image and text multimodal data, and then removes duplicate, null, and outlier values ​​from specific data; for highways in a specific area, for the defined decision interval [T]... k-1 ,T k ), has V ij k = {V ijv k , v=1, 2, …, l} represents the road condition media frame image data or road network gantry text data captured by highway cameras within the decision interval; The high-security collaborative training module performs collaborative training based on preprocessed multimodal data, iteratively trains the model using federated learning and homomorphic encryption technology, securely shares data, and generates an output model M after completing distributed training. The integrated decision optimization module performs decision-level fusion and output optimization of multimodal data based on the output model M generated by the high-security collaborative training module and the interval optimization algorithm, and generates the congestion level output.

Images