Multi-vehicle dam damage identification method and device based on weight adaptation

Abstract

The present invention discloses a multi-vehicle dam damage identification method and device based on weight adaptation. Through adaptive weight calculation and model hierarchical hybrid technology, multiple unmanned vehicles are aggregated to identify and detect damage to dams caused by natural disasters such as earthquakes, and an adaptive damage identification model (i.e., personalized model) is provided for each unmanned vehicle, while ensuring the accuracy of damage identification of the unmanned vehicle. Adaptive weight calculation updates the hierarchical weight value according to the model parameters on each vehicle through the chain derivation rule to obtain the optimal hybrid weight of each vehicle model; the hierarchical hybrid weight is used to hierarchically mix the local model on the vehicle and the intermediate version of the personalized model (i.e., the intermediate model), dynamically adjust the balance between the shared data between all vehicles captured by the global model and the local data learned by the local model, and obtain a personalized model adapted to each vehicle, thereby ensuring the accuracy of damage identification.

Description

Technical Field

[0001] The invention relates to a multi-vehicle dam damage identification method and device based on weight adaptation, belonging to the technical field of image recognition and classification. Background Art

[0002] With the rapid development of information technology and intelligence, various types of vehicles (such as drones, cameras, rail robots, etc.) have been widely used in various fields. These large numbers of unmanned vehicles have collected massive amounts of data, forming a huge distributed Internet of Things. Since the data collected by various vehicles are relatively scattered and highly independent, and the data collected by different vehicles are not independent and identically distributed, it is difficult for a single global model to be applicable to all types of vehicles. Federated learning uses the data of each client to train the local model, and then uploads the model parameters or model gradient updates to the parameter server. The parameter server aggregates the received local model or local model gradient to form a new global model. The client linearly mixes the global model with the local model, which not only takes into account the distribution of local data but also learns global shared knowledge from the global model, making full use of the data of all clients. While protecting the privacy and security of local data, it can train a model suitable for the local data of each client through the collaboration of multiple clients.

[0003] For the identification and detection of damage to dams caused by natural disasters such as earthquakes, due to the large geographical scope and special location of the identification area, each unmanned vehicle is often used to handle damage detection in different areas and of different types. These data are highly independent. In this data heterogeneous environment, it is difficult to process all image data through centralized data processing and analysis with a single recognition model. At the same time, in emergency conditions such as natural disasters, the communication of each vehicle is usually affected, requiring each vehicle to complete the identification and detection of various damages locally as much as possible to reduce the occupation of communication resources. In the above scenario, it is a good solution to equip each unmanned vehicle with a recognition model adapted to its damage data. However, in federated learning, due to the different distribution of client data, the degree of difference between the local model of each client and the global model is not the same, and the local model and the global model are constantly changing in each round of federated training, and the weights when the model is mixed are difficult to determine. At the same time, there are differences between model levels. Mixing the entire model as a whole ignores the hierarchical differences of the model and affects the accuracy of the personalized model. Summary of the invention

[0004] Purpose of the invention: In the identification and detection of damage to dams caused by natural disasters such as earthquakes, in order to address the problem of low accuracy of personalized models caused by data heterogeneity and coarse-grained model mixing in federated learning, the present invention provides a multi-vehicle dam damage identification method and device based on weight adaptation based on adaptive weight calculation and model layered hybrid technology. The method and device can identify and detect damage to dams caused by natural disasters such as earthquakes by aggregating multiple unmanned vehicles, provide an adaptive damage identification model for each unmanned vehicle, and ensure the accuracy of damage identification of the unmanned vehicle at the same time.

[0005] Technical solution: A multi-vehicle dam damage identification method based on weight adaptation. This method uses adaptive weight calculation and model layering hybrid technology to aggregate multiple unmanned vehicles to identify and detect damage to dams caused by natural disasters such as earthquakes, provide an adaptive damage identification model for each unmanned vehicle, and ensure the accuracy of damage identification of unmanned vehicles.

[0006] Adaptive weight calculation updates the layered weight values ​​based on the model parameters obtained during training through the chain rule to find the optimal hybrid weights for each vehicle model;

[0007] Model hierarchical mixing uses hierarchical mixing weights to perform hierarchical mixing of local models and intermediate models, dynamically adjusting the balance between the shared knowledge among all vehicles captured by the global model and the local knowledge learned by the intermediate model, thereby ensuring the accuracy of local model recognition.

[0008] The multi-vehicle dam damage recognition method based on weight adaptation uses the damage image data corresponding to the inspection drone, fixed-point camera, rail robot and wheeled robot, so that the local model trained on each unmanned vehicle is adapted to its damage image data; each vehicle stores three models, namely: the local model, the intermediate version of the personalized model (hereinafter collectively referred to as the intermediate model) and the personalized model, including the following steps:

[0009] Step 1) Based on the dam post-disaster damage image data obtained by multiple vehicles, the dam post-disaster damage recognition model is trained through the federated learning algorithm; at the beginning of each round of federated training, the central control room server randomly selects N vehicles from the vehicle cluster participating in the current training round and uses the global model w of the tth round as the training object. t Send to the selected N vehicles;

[0010] The multiple vehicles include inspection drones, fixed-point cameras, track robots and wheeled robots.

[0011] Step 2) N vehicles receive the global model w t , using the global model w t and local intermediate model parameters and weight Initialize the local model Intermediate Model Weight Parameters Update local model Update intermediate model

[0012] Step 3) After local update, adaptive weight calculation is used to dynamically adjust the model layered hybrid weights through the chain derivation rule

[0013] Step 4) After completing the local model Intermediate Model and model stratified mixture weights After multiple iterations, the weights are mixed based on the model layer Local Model With the intermediate model Perform layered mixing to obtain a personalized model And the local model Upload to the control room server;

[0014] Step 5) The central control room server receives the local models of the N vehicles selected in step 1) to participate in this round of federated training. Parameters, aggregate the received local model parameters according to the federated averaging method to form a new global model;

[0015] Repeat steps 1) to 5) until the specified communication round T is reached or the dam damage identification model of each vehicle converges, and finally output the personalized model set of N vehicles That is, the personalized dam damage identification model of each vehicle. Each vehicle uses the corresponding dam damage identification model to identify the dam damage on the collected dam post-disaster damage image data.

[0016] Furthermore, the specific steps of updating the intermediate model in step 2) are as follows:

[0017] Intermediate Model Does not participate in global aggregation, so when updating the intermediate model When , the cross entropy loss function is used:

[0018]

[0019] in is the cross entropy loss function f CF The stochastic gradient of is the personalized model of vehicle i in the tth round of communication, is the gradient of the cross entropy loss function with respect to v, and η is the learning rate.

[0020] Furthermore, the specific steps of the adaptive weight calculation in step 3) are as follows:

[0021] When training a personalized model, it is necessary to make a trade-off between the local model and the intermediate model based on the weight parameter. The update of the weight coefficient is based on the difference between the local model and the intermediate model and the personalized model. The gradient of can be updated during training according to the chain derivation rule of the function The updated formula is as follows:

[0022]

[0023] in is the layer mixing weight of vehicle i in the tth round of communication, Represents the stochastic gradient of the cross entropy loss function for the personalization model.

[0024] Furthermore, the specific steps of weight perception in step 4) are as follows:

[0025] Through hierarchical representation, local models Intermediate Model and personalized models It can be expressed as:

[0026]

[0027]

[0028]

[0029] Where P is the total number of layers in the model, and denote the j-th layer parameters of the local model, intermediate model, and personalized model respectively;

[0030] Personalized Model Obtained by the following formula:

[0031]

[0032] in is a 1×P vector, Intermediate model for model layering and mixing The weight of the jth layer, ⊙ is the Hadamard product, 1 is the sum A unit vector with all elements equal to 1.

[0033] The multi-vehicle dam damage identification device based on weight self-adaptation includes a central control room server and multiple vehicles as clients, the multiple vehicles include drones, fixed-point cameras, track robots and wheeled robots; the multi-vehicle cluster is used to collect inspection image data of the dam area;

[0034] The central control room server selects a part of vehicles to participate in the federated training, and then each vehicle participating in the training receives the global model from the central control room server. Using the received global model, the personalized model and weight parameters saved in the previous round of training, the local model, intermediate model and weight parameters of this round of communication are initialized respectively;

[0035] Each unmanned vehicle uses the cross entropy loss function to update the local model and the intermediate model. After the local model and the intermediate model on each vehicle are iterated, the local model and the intermediate model are layered and mixed according to the layer weights to obtain a personalized model;

[0036] The central control room server receives local model parameters from the vehicles participating in this round of federated training. Aggregate the received vehicle local model parameters according to the federated averaging method, and update the global model on the central control room server to obtain the global model w of round t+1 t+1 The above process is repeated until the specified communication round is reached or the dam damage identification model of each vehicle converges, and finally the personalized model set of each vehicle is output, which is the personalized dam damage identification model of each vehicle. Each vehicle cluster collects inspection image data of the dam area and uses the personalized dam damage model to identify dam damage.

[0037] The model training and updating process and method of the weight-adaptive multi-vehicle dam damage identification device are the same and will not be repeated here.

[0038] The weight-adaptive multi-vehicle dam damage identification system includes a multi-type acquisition module, a personalized model building model, and a central server module;

[0039] The multi-type acquisition module includes inspection drones, fixed-point cameras, rail-mounted robots and wheeled robots, which are used to collect inspection image data of different areas of the dam. Inspection drones inspect upstream and downstream dam surfaces, fixed-point cameras inspect fixed crack tracking holes and other narrow areas, rail-mounted robots inspect internal corridors, and wheeled robots inspect spacious areas such as the dam top.

[0040] The personalized model building model is used to obtain a personalized model through adaptive weighting and hierarchical mixing after multiple types of vehicles collect data from the inspection area;

[0041] The central server module is used to collect model parameter gradients of various types of vehicles and complete the update of the global model.

[0042] The implementation process of the system is the same as the implementation process of the method, and will not be repeated here.

[0043] Beneficial effects: Compared with the prior art, the present invention provides a multi-vehicle dam damage identification method based on weight adaptation. This method uses adaptive weight calculation and model hierarchical hybrid technology to aggregate multiple unmanned vehicles to identify and detect damage to dams caused by natural disasters such as earthquakes, provide an adaptive damage identification model for each unmanned vehicle, and ensure the accuracy of damage identification of unmanned vehicles; adaptive weight calculation updates the hierarchical weight value through the chain derivation rule according to the model parameters obtained during the training process to find the optimal hybrid weight of each vehicle model; model hierarchical hybrid uses hierarchical hybrid weights to hierarchically mix local models and intermediate models, dynamically adjust the balance between the shared knowledge between all vehicles captured by the global model and the local knowledge learned by the intermediate model, and ensure the accuracy of local model recognition. The present invention solves the problem that the image data collected by each edge vehicle is difficult to upload in large quantities and a single model cannot adapt to the defect data corresponding to each edge vehicle, alleviates the problem of decreased accuracy of the damage identification model caused by the data heterogeneity of each edge vehicle, and ensures the accuracy of the vehicle damage identification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of a device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0047] The weight-adaptive multi-vehicle dam damage identification method, through adaptive weight calculation and model hierarchical hybrid technology, aggregates multiple unmanned vehicles to identify and detect the damage to dams caused by natural disasters such as earthquakes, provides an adaptive damage identification model for each unmanned vehicle, and ensures the accuracy of damage identification of unmanned vehicles.

[0048] like Figure 1 As shown, the weight-adaptive multi-vehicle dam damage identification method includes the following steps:

[0049] Step 1) Based on the dam post-disaster damage image data corresponding to the inspection drones, fixed-point cameras, track robots and wheeled robots, the dam post-disaster damage recognition model is trained through the federated learning algorithm; at the beginning of each round of federated training, the central control room server randomly selects N drones, track robots and wheeled robots from the clusters participating in the current training round, and uses the global model w of the tth round as the training object. tSend to the selected N vehicles;

[0050] Step 2) N vehicles receive the global model w t , using local model parameters and Initialize local model Intermediate Model Weight Parameters Update local model Intermediate Model Does not participate in global aggregation, updates the intermediate model When , the cross entropy loss function is used:

[0051]

[0052] in is the cross entropy loss function f CF The stochastic gradient of is the local personalized model of vehicle i in the tth round of communication, is the gradient of the cross entropy loss function with respect to v, and η is the learning rate.

[0053] Step 3) After the local model is updated, adaptive weight calculation is used to dynamically adjust the model layered hybrid weights through the chain derivation rule. When training the personalized model, it is necessary to weigh the local model and the intermediate model based on the weight parameters. The update of the weight coefficient is based on the difference between the local model and the intermediate model and the personalized model. The gradient of can be updated during training according to the chain derivation rule of the function The updated formula is as follows:

[0054]

[0055] in is the layer mixing weight of vehicle i in the tth round of communication, Represents the stochastic gradient of the cross entropy loss function for the personalization model.

[0056] Step 4) After completing the local model Intermediate Model and model stratified mixture weights After multiple iterations, the weights are mixed based on the model layer Local Model With the intermediate model Perform layered mixing to obtain a personalized model And the local model Upload to the central control room server; through layered representation, the local model on each vehicle Intermediate Model and personalized models It can be expressed as:

[0057]

[0058]

[0059]

[0060] Where P is the total number of layers in the model, and denote the j-th layer parameters of the local model, intermediate model, and personalized model respectively;

[0061] Personalized Model Obtained by the following formula:

[0062]

[0063] in is a 1×P vector, Intermediate model for model layering and mixing The weight of the jth layer, ⊙ is the Hadamard product, 1 is the sum A unit vector with all elements equal to 1.

[0064] Step 5) The central control room server receives the local models of the N vehicles selected in step 1) to participate in this round of federated training. Parameters, aggregate the received local model parameters according to the federated averaging method to form a new global model.

[0065] Repeat steps 1) to 5) until the specified communication round T is reached or the dam damage identification model of each vehicle converges, and finally output the personalized model set That is the personalized dam damage identification model for each vehicle.

[0066] The multi-vehicle dam damage identification device based on weight self-adaptation includes a central control room server and multiple vehicles as clients, the multiple vehicles include drones, fixed-point cameras, track robots and wheeled robots; the multi-vehicle cluster is used to collect inspection image data of the dam area;

[0067] The central control room server selects a part of vehicles to participate in the federated training, and then each vehicle participating in the training receives the global model w from the central control room server. t . Use the received global model, the personalized model and weight parameters saved in the previous round of training to initialize the local model, intermediate model and weight parameters of this round of communication;

[0068] Each unmanned vehicle uses the cross entropy loss function to update the local model and intermediate models After the edge model iteration is completed, the local model and the intermediate model are layered and mixed according to the hierarchical weights to complete the update of the personalized model.

[0069] The central control room server receives the model parameters from the vehicles participating in this round of federated training, aggregates the received model parameters according to the federated averaging method, and updates the global model; repeats the above process until the specified communication round T is reached or the model converges, and finally outputs a personalized model set That is the personalized dam damage identification model for each vehicle.

[0070] like Figure 2 As shown, the weight-adaptive multi-vehicle dam damage identification system includes a multi-type acquisition module, a personalized model building model, and a central server module:

[0071] Among them, the multi-type acquisition modules, including inspection drones, fixed-point cameras, rail-mounted robots and wheeled robots, are used to collect inspection image data from different areas of the dam. Inspection drones inspect the upstream and downstream dam surfaces, fixed-point cameras inspect narrow areas such as fixed crack tracking holes, rail-mounted robots inspect internal corridors, and wheeled robots inspect spacious areas such as the dam top.

[0072] The personalized model building model is used to obtain a personalized model through adaptive weighting and hierarchical mixing after multiple types of vehicles collect data from the inspection area;

[0073] The central server module is used to collect the model parameter gradients of various vehicles and complete the update of the global model.

[0074] Obviously, those skilled in the art should understand that the weight-adaptive multi-vehicle dam damage identification method or weight-adaptive multi-vehicle dam damage identification device of the above-mentioned embodiment of the present invention can be implemented by concentrating on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented with program codes executable by computing devices, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A multi-vehicle dam damage identification method based on weight adaptation, characterized in that: The method uses adaptive weight calculation and model hierarchical hybrid technology to identify and detect the damage to the dam caused by natural disasters by aggregating multiple unmanned vehicles, and provides an adaptive damage identification model for each unmanned vehicle; the method includes the following steps: Step 1) Based on the post-disaster dam damage image data obtained by multiple vehicles, the dam damage recognition model is trained through the federated learning algorithm; at the beginning of each round of federated training, the central control room server randomly selects N vehicles from the vehicle cluster participating in the current training round and uses the global model w of the tth round as the training object. t Send to the selected N vehicles; Step 2) N vehicles receive the global model w t , using the global model w t And the local personality model parameters for round t-1 and weight Initialize the local model Intermediate Model Weight Parameters Update local model Update intermediate model Step 3) After local update, adaptive weight calculation is used to dynamically adjust the model layered hybrid weights through the chain derivation rule Step 4) After completing the local model Intermediate Model and model stratified mixture weights After multiple iterations, the weights are mixed based on the model layer Local Model With the intermediate model Perform layered mixing to obtain a personalized model And the local model Upload to the control room server; Step 5) The central control room server receives the local models of the N vehicles selected in step 1) to participate in this round of federated training. Parameters, aggregate the received local model parameters according to the federated averaging method to form a new global model; The specific steps of weight perception in step 4) are as follows: Through hierarchical representation, local models Intermediate Model and personalized models It can be expressed as: Where P is the total number of layers in the model, and denote the j-th layer parameters of the local model, intermediate model, and personalized model respectively; Personalized Model Obtained by the following formula: in is a 1×P vector, Intermediate model for model layering and mixing The weight of the jth layer, ⊙ is the Hadamard product, 1 is the sum Unit vectors of the same magnitude whose elements are all 1 Repeat steps 1) to 5) until the specified communication round T is reached or the model converges, and finally output a personalized model set of N vehicles. That is, the personalized dam damage identification model of each vehicle; each vehicle uses the corresponding dam damage identification model to identify the dam damage on the collected dam post-disaster damage image data.

2. The weight-adaptive multi-vehicle dam damage identification method according to claim 1 is characterized in that: The multiple vehicles include inspection drones, fixed-point cameras, track robots and wheeled robots.

3. The multi-vehicle dam damage identification method based on weight adaptation according to claim 1 is characterized in that: The specific steps of updating the intermediate model in step 2) are as follows: Intermediate Model Does not participate in global aggregation, updates the intermediate model When , the cross entropy loss function is used: in is the cross entropy loss function f CF The stochastic gradient of is the personalized model of vehicle i in the tth round of communication, is the gradient of the cross entropy loss function with respect to ν, and η is the learning rate.

4. The weight-adaptive multi-vehicle dam damage identification method according to claim 1 is characterized in that: The specific steps of adaptive weight calculation in step 3) are as follows: When training a personalized model, it is necessary to make a trade-off between the local model and the intermediate model based on the weight parameter. The update of the weight coefficient is based on the difference between the local model and the intermediate model and the personalized model. The gradient of can be updated during training according to the chain derivation rule of the function The updated formula is as follows: in is the layer-by-layer mixing weight of vehicle i in the tth round of communication, Represents the stochastic gradient of the cross entropy loss function for the personalization model.

5. A weight-adaptive multi-vehicle dam damage identification device for the method of claim 1, characterized in that: It includes a central control room server and multiple vehicles as clients, including drones, fixed-point cameras, track robots and wheeled robots; a cluster of multiple vehicles is used to collect inspection image data of the dam area; The central control room server selects a part of vehicles to participate in the federated training, and then each vehicle participating in the training receives the global model from the central control room server; using the received global model, the personalized model and weight parameters saved in the previous round of training, the local model, intermediate model and weight parameters of this round of communication are initialized respectively; Each unmanned vehicle uses the cross entropy loss function to update the local model and the intermediate model. After the edge model iteration is completed, the local model and the intermediate model are layered and mixed according to the layer weights to complete the update of the personalized model. After the edge model iteration is completed, the local model and the intermediate model are layered and mixed according to the hierarchical weights to complete the update of the personalized model. The server in the central control room receives the model parameters from the vehicles participating in this round of federated training, aggregates the received model parameters according to the federated averaging method, and updates the global model; until the specified communication round is reached or the model converges, it finally outputs a personalized model set, which is the personalized dam damage identification model for each vehicle; each vehicle cluster collects inspection image data of the dam area and uses the dam damage identification model to identify dam damage.

6. A weight-adaptive multi-vehicle dam damage identification system for implementing the method as claimed in claim 1, characterized in that: It includes multi-type acquisition module, personalized model building model, and central server module; The multi-type acquisition module includes an inspection drone, a fixed-point camera, a rail-mounted robot and a wheeled robot, which are used to collect inspection image data of different areas of the dam. The inspection drone inspects the upstream and downstream dam surfaces, the fixed-point camera inspects narrow areas, the narrow areas include fixed crack tracking holes, the rail-mounted robot inspects internal corridors, and the wheeled robot inspects spacious areas, the spacious areas include the dam top; The personalized model building model is used to obtain a personalized model through adaptive weighting and hierarchical mixing after multiple types of vehicles collect data from the inspection area; The central server module is used to collect model parameter gradients of various types of vehicles and complete the update of the global model.

Images