Mine key facility intelligent visual detection method and system coordinated by unmanned aerial vehicle

Abstract

The application discloses a mine key facility intelligent visual detection method and system cooperating with unmanned aerial vehicles, and the method comprises the following steps: through the cooperation of multiple unmanned aerial vehicles and a server, a double-layer prompt parameter structure is adopted, and through the hierarchical optimization of a flight environment adaptation prompt layer and a facility defect semantic prompt layer, cross-region detection knowledge sharing is realized under the premise of protecting the privacy of regional inspection data; through real-time visual processing and local model training at the unmanned aerial vehicle end, dynamic defect category discovery and detection capability expansion mechanism are realized through the global optimization of federated learning coordination and model distribution at the server end; the unmanned aerial vehicle and the server are cooperatively optimized, the complex environment of the mine is effectively adapted, the detection precision and operation efficiency of the unmanned aerial vehicle mine inspection are significantly improved, and a complete unmanned solution is provided for the maintenance of mine facilities.

Description

Technical Field

[0001] This invention relates to the field of intelligent mine inspection, and in particular to an intelligent visual inspection method and system for key mine facilities using unmanned aerial vehicles (UAVs). Background Technology

[0002] With the large-scale development of the mining industry, the number of mines is increasing daily. Mines encompass various critical facilities such as underground roadways, tailings ponds, belt conveyors, and ventilation shafts. The safety status of these critical facilities directly affects mine production safety and requires regular inspections and condition assessments. Traditional manual inspection methods are not only inefficient and costly, but also pose serious safety hazards such as gas leaks, slope landslides, and dam collapses in dangerous areas such as underground roadways, steep slopes, and tailings dam bodies.

[0003] In recent years, unmanned aerial vehicle (UAV)-based mine inspection technology has gradually emerged. UAVs offer advantages such as maneuverability, comprehensive field of view, and high accessibility, effectively addressing the limitations of traditional inspection methods, such as difficulty in reaching high-risk areas of mines and incomplete inspection coverage. However, the complex and diverse flight environment in mines—dust in open-pit mines, low light and high humidity in underground mines, and drastic changes in light in mountainous mines—all affect image quality, making it difficult for single-vision detection models to maintain stable recognition accuracy. Furthermore, the types of defects in critical mine facilities are complex and exhibit significant regional characteristics, making traditional deep learning models prone to overfitting or insufficient generalization. Moreover, the inspection data from each mine involves core business secrets and security-sensitive information, making centralized processing across regions unsuitable. This hinders the accurate identification of risks in critical mine facilities, resulting in poor timeliness of risk identification and delayed risk response in mining areas. Summary of the Invention

[0004] The main objective of this invention is to provide an intelligent visual inspection method and system for key mining facilities using unmanned aerial vehicles (UAVs), aiming to solve the technical problems of existing technologies being limited by the complex mining environment and low visual inspection accuracy, which makes it impossible to accurately identify the risks existing in key mining facilities, resulting in poor timeliness of risk identification and untimely risk response in mining areas.

[0005] To achieve the above objectives, this invention provides an intelligent visual inspection method for key mining facilities using drone collaboration. The method is applied to the client side of a drone swarm, with each drone communicating with a server. Each drone has a locally deployed CLIP model, which includes an image encoder and a text encoder. The method includes:

[0006] The server receives global parameters, which include global domain adaptability prompt parameters, global defect category semantic distinguishability prompt parameters corresponding to the local mining scenario, and global category center parameters. The global category center parameters are generated by the server based on a global facility defect category set, a template function, and a text encoder. The global facility defect category set is a predefined list of key mining facility defect types.

[0007] Based on the global parameters, the locally learned parameters in the locally deployed CLIP model are updated. The locally learned parameters include local adaptive cue parameters and local defect category semantic discriminative cue parameters. The local adaptive cue parameters are learnable parameters in the CLIP model used to capture environmental features during mine inspection. The local defect category semantic discriminative cue parameters are learnable parameters in the CLIP model used to capture defect category features of key mine facilities. The pre-trained parameters of the image encoder and text encoder in the locally deployed CLIP model remain frozen during the mine inspection process.

[0008] During mine inspection, image data of key mine facilities are collected in real time and input into the CLIP model with updated parameters for visual defect detection, and the target defect category label of key mine facilities is output.

[0009] Optionally, the locally learnable parameters further include local adaptive fusion gating parameters, which are used to dynamically adjust the fusion ratio between the local domain adaptive prompting parameters and the local defect category semantic discriminative prompting parameters. The locally learnable parameters are determined based on historical inspection data or preset values.

[0010] The CLIP model is configured to extract image features from the image data through an image encoder, generate text description information corresponding to each defect type in the global facility defect category set based on the local learnable parameters, encode the text description information through a text encoder to generate multiple text features, take the text features that match the image features as target text features, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text features.

[0011] The CLIP model is further configured to extract image features from the image data through an image encoder, and fuse the local adaptive prompting parameters and the local defect category semantic discriminative prompting parameters based on the local adaptive fusion gating parameters to generate fused prompting parameters;

[0012] The CLIP model is further configured to encode the text description information corresponding to each defect type in the global facility defect category set based on the fusion prompt parameters, and generate multiple text features by encoding the text description information through a text encoder.

[0013] The CLIP model is further configured to calculate the cosine similarity between image features and each text feature, determine the target text feature that matches the image feature based on the cosine similarity, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text feature.

[0014] Optionally, after updating the extended locally learnable parameters in the locally deployed CLIP model based on the global parameters, the method further includes:

[0015] The CLIP model is trained based on the total loss function to optimize the locally learnable parameters in the CLIP model. The total loss function includes a contrast loss term, an intra-class aggregation loss term, and an inter-class separation loss term. The contrast loss term is used to align image features with text features. The intra-class aggregation loss term is used to enhance the feature consistency of image features of the same type of defect. The inter-class separation loss term is used to increase the feature differences of different defects.

[0016] The mathematical expression for the total loss function is:

[0017]

[0018] in, Indicates drone client The total loss term of the total loss function, Indicates the comparison loss term, Representing image features, Representing text features, and These represent the weight hyperparameters, Represents the intra-class aggregate loss term. Indicates the first Defect categories in round training The corresponding global category center parameter, Indicates the defect category, Represents the inter-class separation loss term. This represents the set of defect categories that need to be detected locally by the drone. Represents a set of defect categories In terms of defect categories Different defect categories;

[0019] The intra-class aggregation loss term includes:

[0020]

[0021] in, Represents the local dataset of drones The batch of samples is used for gradient descent training. Represents image data, This represents an image encoder based on the CLIP model. This represents the square of the L2 norm, used to calculate the distance between features;

[0022] The inter-class separation loss term includes:

[0023]

[0024] in, This represents preset boundary hyperparameters, including a preset inter-class center distance threshold. This represents the ReLU function, which is used to constrain the loss to occur only when the distance between classes is insufficient.

[0025] Optionally, after inputting the image data into the parameter-updated CLIP model for defect visual detection and outputting the target defect category label for key mining facilities, the method further includes:

[0026] The local category center parameter is calculated based on the visual inspection results of defects during the inspection process, referring to the following formula:

[0027]

[0028] in, Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Image encoder representing CLIP model For sample images Extracted image features;

[0029] The local update amounts of the local adaptive cue parameters and the local defect category semantic discriminative cue parameters are calculated based on the update results of the extended locally learnable parameters in the locally deployed CLIP model, using the following formulas:

[0030]

[0031]

[0032] in, Indicates drone client The local update amount of the local adaptation hint parameter. This indicates the locale adaptability hint parameter. Indicates the first During the training round, the server sends data to the drone client. Global domain adaptability hint parameters, Indicates the corresponding defect category The local update amount of the local defect category semantic discriminative hint parameter. Indicates the first During the training round, the server sends data to the drone client. corresponding defect categories Global defect category semantic discriminative hint parameters, Indicates drone client Corresponding defect category Local defect category semantic discriminative hint parameters;

[0033] The number of samples corresponding to each local defect category, the local category center parameter, the local update amount of the local domain adaptability prompt parameter, and the local update amount of the local defect category semantic distinguishability prompt parameter are used as upload parameters, and the upload parameters are sent to the server. The server is configured to iteratively aggregate and update the global parameters based on the upload parameters of multiple drone clients.

[0034] Furthermore, to achieve the above objectives, this invention also proposes an intelligent visual inspection method for key mining facilities using drone collaboration. The method is applied to a server, which communicates with each drone in the drone cluster. The server locally deploys a CLIP model, which includes a text encoder. The method includes:

[0035] The global defect category semantic discriminative hint parameters corresponding to each defect category of key mining facilities are initialized using a Gaussian distribution;

[0036] The global domain adaptive prompting parameters are initialized based on the common characteristics of the mine flight environment;

[0037] Based on the initialized global defect category semantic discriminative hint parameters and global domain adaptive hint parameters, the global facility defect category set is encoded into global category center parameters through template functions and text encoders;

[0038] The initialized global defect category semantic discriminative hint parameters, global domain adaptive hint parameters, and global category center parameters are used as global parameters, and the global parameters are sent to each UAV in the UAV cluster;

[0039] The system receives upload parameters sent by the drone, updates the global parameters based on the upload parameters, and sends the updated global parameters to each drone.

[0040] Optionally, receiving the upload parameters sent by the drone, updating the global parameters based on the upload parameters, and sending the updated global parameters to each drone includes:

[0041] Receive the upload parameters sent by the drone, and calculate the domain adaptability aggregation weight of each drone based on the upload parameters, referring to the following formula:

[0042]

[0043] in, Indicates drone client Domain-adaptive aggregation weights, Indicates drone The number of inspection samples of all mining facilities in the collected local dataset. Indicates the first The set of participating clients in the round of federated training. Indicates the first The sum of the total number of samples from all clients participating in the training round;

[0044] The local update amount of the local domain adaptability hint parameter in the uploaded parameters of each UAV is aggregated based on the domain adaptability aggregation weight, and the global domain adaptability hint parameter is updated based on the aggregation result, referring to the following formula:

[0045]

[0046] in, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates drone client The local update amount of the local adaptation hint parameter;

[0047] The semantic discriminative aggregation weight of each drone is determined based on the uploaded parameters, referring to the following formula:

[0048]

[0049] Among them, it represents the drone client. Semantic discriminative aggregation weights, Indicates drone client The local dataset belongs to the defect category The number of samples, This represents the feature consistency weight term. The hyperparameter representing the sensitivity to consistency. Indicates drone client Corresponding category Local category center , and the Wheel category Global category center The squared L2 norm between them is used to represent the characteristic distance between the two. The sign for proportionality;

[0050] The local update amount of the local defect category semantic discriminative hint parameter in the uploaded parameters of each UAV is aggregated based on the semantic discriminative aggregation weight, and the global defect category semantic discriminative hint parameter is updated based on the aggregation result, as shown in the following formula:

[0051]

[0052] in, Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates drone client The local update amount of the local defect category semantic discriminative hint parameter;

[0053] Update the global category center parameter using the following formula:

[0054]

[0055]

[0056] in, It is the momentum coefficient, used to control the degree to which historical center information is retained. Indicates the first The global category center parameter of the wheel, Indicates the first The global category center parameter of the wheel, Indicates the first The local category-weighted average of all participating clients. Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Indicates participation category Aggregates all drone clients;

[0057] The updated global category center parameters, global domain adaptability hint parameters, and global defect category semantic discriminative hint parameters are sent to each UAV.

[0058] Optionally, after updating the global parameters based on the uploaded parameters and sending the updated global parameters to each drone, the process further includes:

[0059] Based on the uploaded parameters of each drone, the performance of each drone is evaluated to obtain the cross-domain adaptability index of each drone, referring to the following formula:

[0060]

[0061] in, Indicates drone client Cross-domain adaptability indicators Indicates the defect category The corresponding global category center parameter, Indicates drone client The number of defect categories that are detected locally;

[0062] The learning rate of each UAV is determined based on the aforementioned cross-domain adaptability index, referring to the following formula:

[0063]

[0064] in, Indicates the first Wheel of drone client The learning rate Indicates the basic learning rate. Indicates the adjustment factor;

[0065] The learning rate and the updated global parameters are sent to each UAV. The UAV is configured to use the learning rate as the step size coefficient of the gradient of the local total loss function, and update the local learnable parameters based on the step size coefficient and the updated global parameters.

[0066] Furthermore, the present invention also proposes a drone, the drone comprising: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor, the drone-coordinated intelligent visual inspection program for key mining facilities being configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described above.

[0067] Furthermore, the present invention also proposes a server, the server comprising: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor, the drone-coordinated intelligent visual inspection program for key mining facilities being configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described above.

[0068] Furthermore, this invention also proposes a drone-assisted intelligent visual inspection system for key mining facilities. The drone-assisted intelligent visual inspection system for key mining facilities includes multiple drones and a server, with each drone communicating with the server. The drone-assisted intelligent visual inspection system for key mining facilities is configured to implement the drone-assisted intelligent visual inspection method for key mining facilities as described above.

[0069] This invention utilizes a collaborative operation between multiple drones and a server, employing a two-tiered prompting parameter structure. Through layered optimization of the flight environment adaptation prompting layer and the facility defect semantic prompting layer, it achieves cross-regional detection knowledge sharing while protecting the privacy of inspection data in each region. The drones perform real-time visual processing and local model training, while the server achieves a dynamic defect category discovery and detection capability expansion mechanism through globally optimized federated learning coordination and model distribution. This collaborative optimization between drones and the server ensures the model's generalization ability by providing global baseline parameters, while the drones adapt to specific scenarios through locally learnable parameters. Combined with the cross-modal matching advantages of the CLIP model, it effectively mines facility defect features in complex mining environments, significantly improving the accuracy and multi-domain adaptability of critical mine facility defect detection. Simultaneously, the automated inspection and detection process greatly reduces manual intervention, lowering the labor costs and safety risks of mine inspections, and improving inspection efficiency and detection reliability. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a schematic diagram of the structure of an intelligent visual inspection device for key mining facilities in collaboration with unmanned aerial vehicles (UAVs) in the hardware operating environment of the embodiment of the present invention.

[0072] Figure 2 This is a flowchart illustrating the first embodiment of the intelligent visual inspection method for key mining facilities using drone collaboration according to the present invention.

[0073] Figure 3This is a flowchart illustrating the second embodiment of the intelligent visual inspection method for key mining facilities using drone collaboration according to the present invention.

[0074] Figure 4 This is a schematic diagram of the framework of the first embodiment of the UAV-coordinated intelligent visual inspection system for key mining facilities of the present invention.

[0075] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0076] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0077] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of an intelligent visual inspection device for key mining facilities using drones in collaboration with the hardware operating environment involved in the embodiments of the present invention.

[0078] like Figure 1 As shown, the intelligent visual inspection equipment for key mining facilities in collaboration with the UAV may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage system independent of the aforementioned processor 1001.

[0079] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on intelligent visual inspection equipment for mine critical facilities in collaboration with drones, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] like Figure 1As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and an intelligent visual inspection program for key mining facilities in collaboration with unmanned aerial vehicles (UAVs).

[0081] exist Figure 1 In the UAV-coordinated intelligent visual inspection equipment for key mining facilities shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the UAV-coordinated intelligent visual inspection equipment for key mining facilities of the present invention can be set in the UAV-coordinated intelligent visual inspection equipment for key mining facilities. The UAV-coordinated intelligent visual inspection equipment for key mining facilities calls the UAV-coordinated intelligent visual inspection program for key mining facilities stored in the memory 1005 through the processor 1001 and executes the UAV-coordinated intelligent visual inspection method for key mining facilities provided in the embodiments of the present invention.

[0082] This invention provides an intelligent visual inspection method for key mining facilities using unmanned aerial vehicles (UAVs) in collaboration with other technologies. Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the intelligent visual inspection method for key mining facilities using drone collaboration according to the present invention.

[0083] In this embodiment, the intelligent visual inspection method for key mining facilities in collaboration with unmanned aerial vehicles includes the following steps:

[0084] Step S10: Receive global parameters sent by the server.

[0085] It should be noted that this embodiment is applied to the client of drones in a drone swarm. Each drone communicates with the server, and each drone has a CLIP model deployed locally. The CLIP model includes an image encoder and a text encoder.

[0086] It should be understood that the executing entity of this embodiment can be a drone with data processing, network communication, and program execution functions, or a terminal electronic device capable of realizing the above functions. The following description uses a drone as an example to illustrate this embodiment and the subsequent embodiments.

[0087] It should be noted that the global parameters can be a set of baseline parameters generated by the server after aggregating training data from multiple drone clients. These parameters serve as a reference standard for updating the local parameters of the drones. They include global domain adaptive cue parameters, global defect category semantic discriminative cue parameters corresponding to the local mining scenario, and global category center parameters, ensuring the consistency of the optimization direction of each drone model.

[0088] Global domain adaptive cue parameters can be learnable parameters that the server learns for common mine inspection environments (such as dust in open-pit mines, low light in underground mines, and dampness in tailings ponds), and are used to provide basic environmental feature references for drones to adapt to specific local mine environments.

[0089] The semantic discriminative hint parameters for global defect categories corresponding to local mining scenarios can be parameters generated by the server in combination with the key facility types of local mines (such as slopes, rollers, dams, etc.) and based on common defects of various facilities. The core function is to strengthen the semantic differences between different defect categories and help the model distinguish similar defects.

[0090] It should be noted that the global category center parameter is generated by the server based on the global facility defect category set, template function, and text encoder. The global facility defect category set is a predefined list of defect types for key mine facilities. The template function is used to convert "defect category label + prompt parameter features" into standardized text statements that the CLIP model can recognize. The output format needs to be adapted to the semantic encoding logic of the text encoder. For example, "slope crack" can be fused with domain-adaptive prompt parameters to generate "slope crack image under open-pit mine dust environment".

[0091] The global category center parameter can be obtained by the server using a global set of facility defect categories as a basis, fusing defect categories and prompt parameters through a template function to generate a standardized text description, and then encoding it through a text encoder to obtain the feature reference vector of each type of defect. This is the core reference for subsequent image and text feature matching.

[0092] The global facility defect category set can be a list of defect types to be detected defined in advance according to the mine safety production specifications and the operation and maintenance requirements of key facilities, such as "slope cracks", "roller damage", "dam leakage", "support deformation", etc., which clarifies the target scope of defect detection.

[0093] It is understood that this embodiment designs two levels of Prompt vectors to model feature information at different levels, including Domain-adaptive Prompt and Semantic-discriminative Prompt for critical facility defect categories.

[0094] Among them, the domain-adaptive prompt (i.e., the domain-adaptive prompt parameter) is a vector characterizing the macroscopic features of the local flight environment of the client (the drone performing the inspection task). It is used to capture the "common environmental features" in the mine inspection scenario and solve the imaging differences caused by different mine flight environments. The adapted mine scene features include environmental factors such as dust / blasting dust in open-pit mines, low light / high humidity in underground mines, and drastic changes in light / shadow occlusion in mountain mines, as well as common information such as drone shooting angle and background texture. Through this prompt, the model can quickly adapt to the local mine flight environment and reduce the impact of environmental interference on imaging quality and detection accuracy.

[0095] Domain adaptability hint parameters are defined as follows , indicating the client The local adaptation Prompt vector. This vector primarily captures the client's... Dataset The overall, macroscopic feature distribution includes common client-side features such as lighting conditions, shooting angle, and background texture. Among them The dimension of the Prompt vector and the CLIP model feature space (i.e., d is the dimension of the feature space). The server maintains the corresponding global version. .

[0096] The semantic discriminative prompt for defect categories (i.e., the semantic discriminative prompt parameter for defect categories) is a set of vectors that correspond one-to-one with the defect categories of mining facilities. Each vector specifically represents the subtle intrinsic features of a certain type of defect, used to achieve accurate differentiation between different defect categories. It focuses on the "specific features at the defect category level," addressing the problem of insufficient model generalization ability caused by the complexity of defect types and significant regional characteristics in mining facilities. Adapted mine defect types include: slope cracks / stripping in open-pit mines, belt conveyor deviation / roller damage, surrounding rock deformation / support damage in underground roadways, and tailings dam leakage / collapse, etc.—each defect category corresponds to a dedicated semantic discriminative prompt, ensuring that the model can accurately capture the unique features of various defects and avoid overfitting or category confusion.

[0097] The semantic discriminative hint parameter for defect categories is defined as follows: , representing the set of global semantic discriminative Prompt vectors. Is it related to a specific defect category? The associated Prompt vector is used to characterize the intrinsic, subtle characteristics of the category, enabling effective differentiation from other categories. For the set of all defect categories, Indicates the total number of categories. (Client) Maintain and update its subset of categories during local training. Corresponding semantic distinguishability Prompt .

[0098] In the specific implementation, the participants are the first Round training client Receive the following global parameters from the server:

[0099] 1. Current Global Domain Adaptive Prompt: ;

[0100] 2. Prompt for global semantic distinctiveness corresponding to local categories: ,in It is a client The collection of category tags owned locally;

[0101] 3. The global category center corresponding to the local category: .

[0102] Step S20: Update the extended locally learnable parameters in the locally deployed CLIP model based on the global parameters.

[0103] It should be noted that the locally learnable parameters include local adaptive cue parameters and local defect category semantic discriminative cue parameters. The local adaptive cue parameters are learnable parameters in the CLIP model used to capture environmental features during mine inspections, and the local defect category semantic discriminative cue parameters are learnable parameters in the CLIP model used to capture defect category features of key mine facilities. The pre-trained parameters of the image encoder and text encoder in the locally deployed CLIP model remain frozen during mine inspections.

[0104] In the specific implementation, the drone client initializes local parameters based on global parameters:

[0105] Local Adaptive Prompt Tuning:

[0106]

[0107] During local training, optimization is performed using gradient descent. The local adjustment amount is obtained:

[0108]

[0109] Local semantic discriminative Prompt update: for each :

[0110]

[0111] Optimization via gradient descent Get the local update count:

[0112]

[0113] It is understandable that this embodiment updates the local learnable parameters, enabling the UAV model to accurately adapt to the unique environment and defect characteristics of the local mine while inheriting the generalization ability of global parameters, thus achieving a dual optimization of "global generalization + local adaptation". Freezing the encoder pre-training parameters effectively reduces the computing power consumption of local training, adapts to the embedded hardware resource limitations of the UAV, and ensures the stability and reliability of feature extraction.

[0114] Furthermore, in order to effectively and dynamically adjust the fusion ratio of information at different levels and achieve more refined personalized feature generation, in one embodiment, the local learnable parameter further includes a local adaptive fusion gating parameter. The local adaptive fusion gating parameter is used to dynamically adjust the fusion ratio between the local domain adaptive prompting parameter and the local defect category semantic distinguishing prompting parameter. The local learnable parameter is determined based on historical inspection data or preset values.

[0115] It should be noted that the CLIP model is configured to extract image features from the image data through an image encoder, generate text description information corresponding to each defect type in the global facility defect category set based on the locally learnable parameters, encode the text description information through a text encoder to generate multiple text features, take the text features that match the image features as target text features, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text features.

[0116] The CLIP model is further configured to extract image features from the image data through an image encoder, and fuse the local adaptive prompting parameters and the local defect category semantic discriminative prompting parameters based on the local adaptive fusion gating parameters to generate fused prompting parameters;

[0117] The CLIP model is further configured to encode the text description information corresponding to each defect type in the global facility defect category set based on the fusion prompt parameters, and generate multiple text features by encoding the text description information through a text encoder.

[0118] The CLIP model is further configured to calculate the cosine similarity between image features and each text feature, determine the target text feature that matches the image feature based on the cosine similarity, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text feature.

[0119] It should be noted that pre-trained parameter freezing means that the original pre-trained weights of the image encoder and text encoder in the CLIP model are kept fixed and do not participate in the backpropagation process of local parameter updates. The core purpose is to avoid the degradation of the encoder's general feature extraction capability due to training on a small amount of local data.

[0120] The local adaptive fusion gating parameter is an important component of the locally learnable parameters. It is used to dynamically adjust the fusion ratio between the local adaptive cue parameters and the local defect category semantic discriminative cue parameters, so that the model can flexibly adjust the weights of the two types of features according to different inspection environments and defect features. Its initial value can be determined based on historical inspection data (such as the optimal fusion ratio value of similar scenarios in the past) or a preset value (such as 0.5), and will be continuously optimized with local training.

[0121] It is understandable that this embodiment introduces an adaptive cross-hierarchical fusion gate, i.e., an adaptive fusion gate parameter, to dynamically adjust the fusion ratio of "domain adaptability prompt (common environmental features)" and "semantic discriminativeness prompt (defect category characteristics)" in complex mining scenarios. This achieves a personalized balance between "environmental adaptation" and "accurate defect identification" in complex mining scenarios, and ultimately improves the model's adaptability to heterogeneous mining environments and diverse defects.

[0122] Introducing Adaptive Fusion Gated Scalar This parameter is provided by the client. For category Acquired through local learning. Dynamic adjustment of gating parameters and adaptive domain prompt. For a specific category The degree of influence of feature generation. When When the value is close to 1, it indicates a strong reliance on common client characteristics; when it is close to 0, it indicates a greater emphasis on category-specific characteristics.

[0123] For drone clients Defect categories on Its text description Generate in the following way:

[0124]

[0125] in, It is a predefined template function that receives category labels. The combined Prompt vector is used to generate a text description for the input text encoder.

[0126] In the specific implementation, local adaptive fusion gating initialization: for each defect type It can be determined based on historical inspection data or preset values:

[0127]

[0128] in, These are locally adaptive fusion gated parameters, which are trainable parameters and optimized together with the Prompt using gradient descent. Indicates the first Local adaptive fusion gating parameters of the previous round (i.e., the previous round).

[0129] Furthermore, in order to accurately align image features and text features, and to accurately match the defect types corresponding to the facility image features, in one embodiment, after step S20 above, the following may be included:

[0130] Step S201: Train the CLIP model based on the total loss function to optimize the locally learnable parameters in the CLIP model.

[0131] It should be noted that the total loss function includes a contrast loss term, an intra-class aggregation loss term, and an inter-class separation loss term. The contrast loss term is used to align image features with text features, the intra-class aggregation loss term is used to enhance the feature consistency of image features with the same type of defects, and the inter-class separation loss term is used to increase the feature differences between different defects.

[0132] It should be noted that the total loss function is a comprehensive loss metric used to guide the optimization of locally learnable parameters. By fusing contrastive loss terms, intra-class aggregation loss terms, and inter-class separation loss terms, it achieves multi-dimensional constraints on image-text feature alignment, aggregation of similar features, and separation of dissimilar features.

[0133] The contrastive loss term is a loss component used to constrain the alignment of image features with corresponding text features. Its core objective is to increase the similarity between image and text features of the same defect category, reduce the similarity between different defect categories, and improve cross-modal matching accuracy.

[0134] The intra-class aggregation loss term is used to strengthen the consistency of features of defects of the same type. By constraining local sample features to move closer to the global class center, it reduces the specific impact of local data of a single UAV and improves the generalization ability of the model.

[0135] The inter-class separation loss term is used to increase the difference in features between different defects. By setting a boundary threshold to constrain the distance between the centers of different defect categories, it avoids category confusion caused by similar visual features of defects and improves the accuracy of defect differentiation.

[0136] The mathematical expression for the total loss function is:

[0137]

[0138] in, Indicates drone client The total loss term of the total loss function, This indicates a comparative loss term (such as InfoNCE loss). Representing image features, Representing text features, and These represent the weight hyperparameters, Represents the intra-class aggregate loss term. Indicates the first Defect categories in round training The corresponding global category center parameter, Indicates the defect category, Represents the inter-class separation loss term. This represents the set of defect categories that need to be detected locally by the drone. Represents a set of defect categories In terms of defect categories Different defect categories;

[0139] The intra-class aggregation loss term includes:

[0140]

[0141] in, Represents the local dataset of drones The batch of samples is used for gradient descent training. Represents image data, This represents an image encoder based on the CLIP model. This represents the square of the L2 norm, used to calculate the distance between features.

[0142] The average form of the intra-class aggregate loss term on mini-batch is:

[0143]

[0144] in It comes from a local dataset of drones. A mini-batch (small batch of samples used for gradient descent training).

[0145] The inter-class separation loss term includes:

[0146]

[0147] in, This represents preset boundary hyperparameters, including a preset inter-class center distance threshold. This represents the ReLU function, which is used to constrain the loss to occur only when the distance between classes is insufficient.

[0148] In some embodiments, the UAV divides the inspection dataset stored locally into a preset ratio (e.g., training set: validation set = 8:2); it uses random sampling to extract mini-batch data (the batch size can be adapted according to the UAV's computing power, such as 16 or 32), and each batch contains image samples of key mining facilities and corresponding defect category labels; it performs standardized preprocessing on the image samples within the batch (consistent with the preprocessing logic in step 3 to ensure uniform data distribution), and loads the local learnable parameters of the corresponding samples (including local adaptive cue parameters, local defect category semantic discriminative cue parameters, and local adaptive fusion gating parameters).

[0149] Based on the calculated total loss value, the gradients of each local learnable parameter are calculated using the backpropagation algorithm. Stochastic gradient descent (SGD) or Adam optimizer is used, with the learning rate updated in step 2 as the step size coefficient, to update the gradients of the local adaptive cue parameters, the local defect category semantic discriminative cue parameters, and the local adaptive fusion gating parameters. During the update process, the pre-trained parameters of the CLIP model image encoder and text encoder are kept frozen, and only the three types of local learnable parameters are iteratively optimized.

[0150] Understandably, this embodiment achieves accurate alignment of image-text features by introducing a contrast loss term, solving the matching bias problem caused by the susceptibility of visual features of defects to interference in complex mining environments, and improving the reliability of cross-modal feature matching. The intra-class aggregation loss term effectively reduces the risk of model overfitting caused by the small amount and uneven distribution of local data from a single UAV by constraining local features to converge towards the global center, ensuring the consistency of detection standards among different UAVs. The inter-class separation loss term increases the feature differences between different defect categories, alleviating the problem of category confusion between similar defects in mines (such as "slope cracks" and "dam leakage"), and improving the accuracy of defect classification. The total loss function, which is a weighted fusion of multiple loss terms, achieves the synergistic optimization of "feature alignment - intra-class aggregation - inter-class separation", enabling local learnable parameters to adapt to local scenarios while taking into account global generalization ability and category discrimination ability, further improving the overall performance of defect detection on the UAV side.

[0151] Step S30: During the mine inspection process, image data of key mine facilities are collected in real time, and the image data is input into the CLIP model with updated parameters for visual defect detection, and the target defect category label of the key mine facilities is output.

[0152] It should be noted that key mining facilities refer to facilities and equipment that play a core supporting role in the production and operation of a mine and directly affect safe production, such as open-pit mine slopes, underground supports, belt conveyor rollers, and tailings dam bodies.

[0153] Image data can be visual data containing appearance information of key mining facilities collected by drones through industrial cameras. It serves as the raw input for defect detection and must possess clear facility details and defect features. Defect visual detection refers to the process of identifying and classifying defects in key mining facilities in images using computer vision technology. In this embodiment, it is achieved through cross-modal matching of the CLIP model.

[0154] The target defect category label can be the category identifier corresponding to the defect in the image determined by the model through feature matching, that is, the specific defect type in the global facility defect category set (such as "slope crack"), and is the final output result of defect detection.

[0155] In practical implementation, the local dataset for drone-based maintenance of critical mining facilities inspections is used. Each sample in ,in It is image data. This corresponds to the defect type label. Image feature extraction is performed on the image data:

[0156]

[0157] in, This represents the image encoder of the CLIP model (pre-trained parameters frozen, used only for extracting image features). Representing image features, Represents a d-dimensional feature dimension space;

[0158] Text descriptions are generated for each defect type in the global facility defect category set: Text descriptions are generated using the updated parameters.

[0159]

[0160] in, This represents a template function used to convert defect labels and fusion prompts into textual descriptions that CLIP can understand. This indicates the locale adaptability hint parameter. This indicates a semantically discriminative parameter for local defect categories. This represents textual description information. Indicates the defect category, Indicates local adaptive fusion gating parameters (for UAVs targeting defect categories) The learned scalar parameters are used to adjust the fusion ratio of the two types of prompts.

[0161] Text feature extraction is performed on the generated text description information:

[0162]

[0163] in, This represents the text encoder of the CLIP model (pre-trained parameters frozen, used only for extracting text features). Representing text features;

[0164] It is understandable that this embodiment realizes an automated inspection image acquisition and defect detection process, which greatly reduces the participation of manual inspection, lowers the labor cost and personnel safety risks of mine inspection (avoiding personnel entering high-risk areas); image preprocessing and invalid image filtering improve the quality of input model data, providing a guarantee for subsequent detection accuracy; real-time acquisition and detection adapt to the timeliness requirements of mine inspection, and can quickly detect facility defects and issue early warnings.

[0165] Furthermore, in order to promote continuous iterative optimization of global parameters and further improve the collaborative detection capability and multi-domain generalization performance of the entire UAV swarm, in one embodiment, after step S30 above, the following is also included:

[0166] Step S301: Calculate the local category center parameters based on the visual inspection results of defects during the inspection process.

[0167] In practical implementation, the drone client Calculate each category it possesses The local average image features are defined by the following formula:

[0168]

[0169] in, Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Image encoder representing CLIP model For sample images Extracted image features.

[0170] Step S302: Calculate the local update amount of the local adaptive hint parameters and the local update amount of the local defect category semantic discriminative hint parameters based on the update results of the extended locally learnable parameters in the locally deployed CLIP model, referring to the following formula:

[0171]

[0172]

[0173] in, Indicates drone client The local update amount of the local adaptation hint parameter. This indicates the locale adaptability hint parameter. Indicates the first During the training round, the server sends data to the drone client. Global domain adaptability hint parameters, Indicates the corresponding defect category The local update amount of the local defect category semantic discriminative hint parameter. Indicates the first During the training round, the server sends data to the drone client. corresponding defect categories Global defect category semantic discriminative hint parameters, Indicates drone client Corresponding defect category Local defect category semantic discriminative hint parameters;

[0174] Step S303: The number of samples corresponding to each local defect category, the local category center parameter, the local update amount of the local domain adaptability prompt parameter, and the local update amount of the local defect category semantic discriminative prompt parameter are used as upload parameters, and the upload parameters are sent to the server. The server is configured to iteratively aggregate and update the global parameters based on the upload parameters of multiple drone clients.

[0175] In practical implementation, the drone client Upload the following parameters to the server:

[0176] Local Prompt update count: and ;

[0177] Local Category Center: ;

[0178] Corresponding sample size: .

[0179] In this embodiment, the UAV achieves collaborative optimization with the server through a full-process design that receives global parameters, updates local parameters, and collects and detects data in real time. By combining client commonalities (domain adaptability) and category characteristics (semantic distinguishability) in the Prompt design, it effectively adapts to the data heterogeneity problem in federated learning. The server provides global baseline parameters to ensure the model's generalization ability, while the UAV adapts to specific scenarios through locally learnable parameters. Combined with the cross-modal matching advantage of the CLIP model, it effectively mines the facility defect characteristics in complex mining environments, significantly improving the accuracy and multi-domain adaptability of key mine facility defect detection. At the same time, the automated inspection and detection process greatly reduces manual intervention, lowers the labor costs and safety risks of mine inspection, and improves inspection efficiency and detection reliability.

[0180] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the intelligent visual inspection method for key mining facilities using drone collaboration according to the present invention.

[0181] Based on the first embodiment described above, in this embodiment the method is applied to a server, which communicates with each drone in the drone cluster. The server locally deploys a CLIP model, which includes a text encoder. The method further includes:

[0182] Step S1: Initialize the global defect category semantic discriminative hint parameters corresponding to each defect category of the key facilities in the mine using a Gaussian distribution.

[0183] It should be understood that the executing entity of this embodiment can be a server with data processing, network communication, and program execution functions, or a terminal electronic device capable of implementing the above functions. The following description uses a server as an example to illustrate this embodiment and the subsequent embodiments.

[0184] It should be noted that the global domain adaptability cue parameters are learnable parameters initialized by the server and used to adapt to the general flight environment of mines, providing a unified environmental characteristic benchmark for optimizing the local domain adaptability parameters of each UAV.

[0185] Understandably, during the system initialization phase, the server first loads the pre-trained CLIP model, which contains the image encoder. and text encoder The parameters of these encoders remain frozen during federated training and do not participate in parameter updates. Represents image space, Represents the text space.

[0186] In the specific implementation, for the initial set of known facility defect categories Each defect category Its global semantic discriminative prompt is initialized using a Gaussian distribution:

[0187]

[0188] in Indicate category In the initial semantic discriminative prompt of round 0, This indicates that the mean is 0 and the covariance is... Gaussian distribution, yes The identity matrix, It is the initialization variance hyperparameter.

[0189] Step S2: Initialize the global domain adaptive prompting parameters based on the common characteristics of the mine flight environment.

[0190] It should be noted that the common characteristics of the mine flight environment refer to the general environmental attributes that all UAV inspections in the mine area must face. These are the core environmental factors that affect the quality of inspection images and the effectiveness of model detection, such as dust, low light, and high humidity.

[0191] It should be noted that the global defect category semantic discriminative hint parameter is a learnable parameter initialized by the server for each type of mine defect, used to enhance the semantic differences between defect categories, and supports each UAV in accurately distinguishing the semantic features of different defects.

[0192] Understandably, global domain adaptive Prompt Initialize to a zero vector or the mean of the initial Prompt for all classes:

[0193]

[0194] Step S3: Based on the initialized global defect category semantic discriminative hint parameters and global domain adaptive hint parameters, the global facility defect category set is encoded into global category center parameters through template functions and text encoders.

[0195] It should be noted that the global category center parameter is a semantic feature baseline vector generated by the server based on the prompt parameters and defect categories. It is the core basis for measuring the rationality of the local category center of the UAV and achieving global feature unification.

[0196] In the specific implementation, the server calculates the initial global category center. , serving as the initial reference points for each category in the feature space:

[0197]

[0198] The category center is generated by the initial global domain adaptability and semantic discriminative prompt through a template function and a text encoder, providing a benchmark for subsequent intra-class aggregation and inter-class separation loss calculations.

[0199] Step S4: Use the initialized global defect category semantic discriminative hint parameters, global domain adaptability hint parameters, and global category center parameters as global parameters, and send the global parameters to each UAV in the UAV cluster.

[0200] In some embodiments, the server retrieves the initialized global defect category semantic distinguishability hint parameters, global domain adaptability hint parameters, and global category center parameters from the database, supplements the global facility defect category set list, and encapsulates it into a global parameter data packet according to a preset format (balancing transmission efficiency and data compression requirements); establishes an encrypted communication link with each drone in the drone cluster through an industrial-grade wireless communication gateway; sequentially sends global parameter data packets according to drone number, and receives reception confirmation signals (including parameter verification codes) from each drone; for drones that fail to confirm or fail verification, a breakpoint resume mechanism is triggered to ensure that all drones receive the global parameters completely; after the sending is completed, a sending log is generated to record the reception status of each drone.

[0201] Step S5: Receive the upload parameters sent by the drone, update the global parameters based on the upload parameters, and send the updated global parameters to each drone.

[0202] In some embodiments, a multi-threaded receiving service is enabled to receive uploaded parameter data packets sent by each drone; each data packet is validated and its format parsed to extract core information (number of samples for each defect category, local category center parameter, and update amount of the two types of prompt parameters), and invalid or abnormal data (such as data with negative sample numbers or mismatched parameter dimensions) is filtered out; an aggregation weight is calculated based on the number of defect category samples for each drone; a global weight is calculated based on the total number of samples from the drones, and the global parameters are updated by weighted summation of the update amount of the local adaptive prompt parameters; the updated global parameters are repackaged and sent to each drone to complete one round of global parameter iterative update, while recording the update round and parameter change amount.

[0203] In some embodiments, the server checks if any client has uploaded an entry that does not belong to the current global category set. New category Information. For each new category discovered:

[0204] Identify the set of clients that discovered this new category:

[0205]

[0206] New category semantic distinguishability Prompt initialization: Initialized using the amount of updates uploaded by the client that discovered the category:

[0207]

[0208] New category global center initialization: using the uploaded local center for weighted average:

[0209]

[0210] Global category collection update:

[0211]

[0212] Furthermore, in order to effectively aggregate the uploaded parameters from each drone and improve the rationality of global parameter updates, thereby improving defect detection accuracy, step S5 above may include:

[0213] Step S51: Receive the upload parameters sent by the UAV, and calculate the domain adaptability aggregation weight of each UAV based on the upload parameters;

[0214] Step S52: Aggregate the local update amount of the local domain adaptation prompt parameters in the uploaded parameters of each UAV based on the domain adaptation aggregation weight, and update the global domain adaptation prompt parameters based on the aggregation result;

[0215] Step S53: Determine the semantic discriminative aggregation weight of each UAV based on the uploaded parameters;

[0216] Step S54: Aggregate the local update amount of the local defect category semantic discriminative hint parameter in the uploaded parameters of each UAV based on the semantic discriminative aggregation weight, and update the global defect category semantic discriminative hint parameter based on the aggregation result;

[0217] Step S55: Update the global category center parameters;

[0218] Step S56: Send the updated global category center parameters, global domain adaptability hint parameters, and global defect category semantic discriminative hint parameters to each UAV.

[0219] In the actual implementation, the server calculates for each client. The domain-adaptive aggregation weights are based on their local dataset size:

[0220]

[0221] in, Indicates drone client Domain-adaptive aggregation weights, Indicates drone The number of inspection samples of all mining facilities in the collected local dataset. Indicates the first The set of participating clients in the round of federated training. Indicates the first The sum of the total number of samples from all clients participating in the training round.

[0222] The global domain adaptability prompt is updated based on local adjustments uploaded by the aggregation client.

[0223]

[0224] in, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates drone client The local update amount of the local adaptation hint parameter.

[0225] For global category set Each category :

[0226] Determine the set of clients participating in the aggregation:

[0227]

[0228] Computing Client Category The semantically discriminative aggregation weight. This weight comprehensively considers data volume and consistency between the local and global centers:

[0229]

[0230] Among them, it represents the drone client. Semantic discriminative aggregation weights, Indicates drone client The local dataset belongs to the defect category The number of samples, This represents the feature consistency weight term. The hyperparameter representing the sensitivity to consistency. Indicates drone client Corresponding category Local category center , and the Wheel category Global category center The squared L2 norm between them is used to represent the characteristic distance between the two. The sign indicates a direct proportion.

[0231] The semantic discriminative aggregation weights are normalized:

[0232]

[0233] Semantic Discrimination Prompt Aggregation Formula:

[0234]

[0235] in, Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates drone client The local update amount of the semantic discriminative hint parameter for the local defect category.

[0236] The global category center is updated smoothly using momentum updates, reducing oscillations during the update process.

[0237]

[0238]

[0239] in, It is the momentum coefficient, used to control the degree to which historical center information is retained. Indicates the first The global category center parameter of the wheel, Indicates the first The global category center parameter of the wheel, Indicates the first The local category-weighted average of all participating clients. Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Indicates participation category Aggregates all drone clients.

[0240] Furthermore, in order to continuously optimize the generalization performance of the system, after step S5 above, the following steps are included:

[0241] Step S5001: Based on the uploaded parameters of each UAV, perform a performance evaluation on each UAV to obtain the cross-domain adaptability index of each UAV.

[0242] Step S5002: Determine the learning rate of each UAV based on the cross-domain adaptability index;

[0243] Step S5003: Send the learning rate and the updated global parameters to each UAV. The UAV is configured to use the learning rate as the step size coefficient of the gradient of the local total loss function, and update the local learnable parameters based on the step size coefficient and the updated global parameters.

[0244] In practice, the server periodically evaluates the difference between the local performance of each client and the global model performance, quantifying cross-domain adaptability through the following metrics:

[0245]

[0246] in, Indicates drone client Cross-domain adaptability indicators Indicates the defect category The corresponding global category center parameter, Indicates drone client The number of defect categories that are detected locally.

[0247] The learning rate is dynamically adjusted based on the client's adaptive evaluation results.

[0248]

[0249] in, Indicates the first Wheel of drone client The learning rate Indicates the basic learning rate. This represents the adjustment factor; clients with low adaptability receive a higher learning rate, accelerating their convergence process.

[0250] This embodiment uses Gaussian distribution and mine environment characteristics for parameter initialization, ensuring the rationality and universality of global parameters. This provides a unified and reliable optimization benchmark for each UAV, avoiding imbalances in collaborative training caused by initial parameter deviations. The generation of global category center parameters combines prompt parameters and scene text descriptions, fully integrating environmental features and defect semantic features, improving the discriminative ability of the global center, and providing accurate reference for subsequent UAV local feature alignment. Aggregated weight calculation based on sample quantity ensures that global parameter updates take into account the sample contribution of different UAVs, avoiding excessive interference from UAVs with small sample sizes, thus improving the rationality of global parameter updates. The application of momentum update strategy reduces global parameter update oscillations and ensures the stability of parameter iteration. The entire "initialization-deployment-update-redeployment" process constructs a collaborative closed loop between the server and UAVs, promoting continuous optimization of global parameters, significantly improving the overall detection consistency and multi-domain generalization ability of the UAV cluster, and providing core technical support for large-scale collaborative inspection in mines.

[0251] Furthermore, this embodiment of the invention also proposes a drone, which includes: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor. The drone-coordinated intelligent visual inspection program for key mining facilities is configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described above.

[0252] Furthermore, this embodiment of the invention also proposes a server, the server comprising: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor, the drone-coordinated intelligent visual inspection program for key mining facilities being configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described above.

[0253] Furthermore, this embodiment of the invention also proposes a computer-readable storage medium storing a drone-assisted intelligent visual inspection program for key mining facilities. When the drone-assisted intelligent visual inspection program for key mining facilities is executed by a processor, it implements the steps of the drone-assisted intelligent visual inspection method for key mining facilities as described above.

[0254] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0255] The aforementioned computer-readable storage medium may be included in the UAV-coordinated intelligent visual inspection equipment for key mining facilities; or it may exist independently and not be assembled into the UAV-coordinated intelligent visual inspection equipment for key mining facilities.

[0256] Furthermore, this invention also proposes a computer program product, including a drone-assisted intelligent visual inspection program for key mining facilities. When the drone-assisted intelligent visual inspection program for key mining facilities is executed by a processor, it implements the steps of the drone-assisted intelligent visual inspection method for key mining facilities as described above.

[0257] The specific implementation of the computer program product of the present invention is basically the same as the embodiments of the above-mentioned intelligent visual inspection method for key mining facilities in collaboration with UAVs, and will not be repeated here.

[0258] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the intelligent visual inspection system for key mining facilities in collaboration with unmanned aerial vehicles (UAVs) of the present invention.

[0259] like Figure 4 As shown in the figure, the intelligent visual inspection system for key mining facilities in collaboration with drones proposed in this embodiment of the invention includes: multiple drones and a server, with each drone communicating with the server.

[0260] The drone 10 is used to receive global parameters sent by the server. The global parameters include global domain adaptability prompt parameters, global defect category semantic distinguishability prompt parameters corresponding to the local mining scenario, and global category center parameters. The global category center parameters are generated by the server based on the global facility defect category set, template function, and text encoder. The global facility defect category set is a predefined list of key mining facility defect types.

[0261] The drone 10 is also used to update the extended locally learnable parameters in the locally deployed CLIP model based on the global parameters. The locally learnable parameters include local adaptive cue parameters and local defect category semantic discriminative cue parameters. The local adaptive cue parameters are learnable parameters in the CLIP model used to capture environmental features during mine inspection. The local defect category semantic discriminative cue parameters are learnable parameters in the CLIP model used to capture defect category features of key mine facilities. The pre-trained parameters of the image encoder and text encoder in the locally deployed CLIP model remain frozen during the mine inspection.

[0262] The drone 10 is also used to collect image data of key facilities in the mine in real time during mine inspection, and input the image data into the CLIP model with updated parameters for visual defect detection, and output the target defect category label of the key facilities in the mine.

[0263] The CLIP model is configured to extract image features from the image data using an image encoder, generate text description information corresponding to each defect type in the global facility defect category set based on the locally learnable parameters, encode the text description information using a text encoder to generate multiple text features, take the text features that match the image features as target text features, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text features.

[0264] The server 20 is used to initialize the global defect category semantic discriminative hint parameters corresponding to each defect category of the key facilities in the mine using a Gaussian distribution;

[0265] The server 20 is also used to initialize global domain adaptive prompt parameters based on the common characteristics of the mine flight environment;

[0266] The server 20 is also used to encode the global facility defect category set into global category center parameters based on the initialized global defect category semantic discriminative hint parameters and global domain adaptive hint parameters through a template function and a text encoder;

[0267] The server 20 is also used to take the initialized global defect category semantic distinguishability hint parameters, global domain adaptability hint parameters and global category center parameters as global parameters, and send the global parameters to each drone in the drone cluster;

[0268] The server 20 is also used to receive upload parameters sent by the drone, update the global parameters based on the upload parameters, and send the updated global parameters to each drone.

[0269] This embodiment utilizes a collaborative operation between multiple UAVs and a server, employing a two-tiered prompting parameter structure. Through layered optimization of the flight environment adaptation prompting layer and the facility defect semantic prompting layer, cross-regional detection knowledge sharing is achieved while protecting the privacy of inspection data in each region. The UAVs perform real-time visual processing and local model training, while the server achieves a dynamic defect category discovery and detection capability expansion mechanism through globally optimized federated learning coordination and model distribution. This enables collaborative optimization between UAVs and the server, with the server providing global baseline parameters to ensure model generalization capabilities. The UAVs adapt to specific scenarios through locally learnable parameters, and combined with the cross-modal matching advantages of the CLIP model, effectively mining facility defect characteristics in complex mining environments. This significantly improves the accuracy and multi-domain adaptability of critical mine facility defect detection. Simultaneously, the automated inspection and detection process greatly reduces manual intervention, lowering the labor costs and safety risks of mine inspections, and improving inspection efficiency and detection reliability.

[0270] The UAV-coordinated intelligent visual inspection system for key mining facilities provided in this application employs the UAV-coordinated intelligent visual inspection method for key mining facilities described in the above embodiments, and can solve the technical problems of UAV-coordinated intelligent visual inspection of key mining facilities. Compared with the prior art, the beneficial effects of the UAV-coordinated intelligent visual inspection system for key mining facilities provided in this application are the same as those of the UAV-coordinated intelligent visual inspection method for key mining facilities provided in the above embodiments, and other technical features of the UAV-coordinated intelligent visual inspection system for key mining facilities are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0271] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0272] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0273] In addition, for technical details not described in detail in this embodiment, please refer to the UAV-coordinated intelligent visual inspection method for key mining facilities provided in any embodiment of the present invention, which will not be repeated here.

[0274] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0275] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0276] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0277] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for intelligent visual inspection of key mining facilities using unmanned aerial vehicles (UAVs) in collaboration, characterized in that, The method is applied to the client of drones in a drone swarm. Each drone communicates with the server, and each drone has a CLIP model deployed locally. The CLIP model includes an image encoder and a text encoder. The method includes: The server receives global parameters, which include global domain adaptability prompt parameters, global defect category semantic distinguishability prompt parameters corresponding to the local mining scenario, and global category center parameters. The global category center parameters are generated by the server based on a global facility defect category set, a template function, and a text encoder. The global facility defect category set is a predefined list of key mining facility defect types. Based on the global parameters, the locally learned parameters in the locally deployed CLIP model are updated. The locally learned parameters include local adaptive cue parameters and local defect category semantic discriminative cue parameters. The local adaptive cue parameters are learnable parameters in the CLIP model used to capture environmental features during mine inspection. The local defect category semantic discriminative cue parameters are learnable parameters in the CLIP model used to capture defect category features of key mine facilities. The pre-trained parameters of the image encoder and text encoder in the locally deployed CLIP model remain frozen during the mine inspection process. During mine inspection, image data of key mine facilities are collected in real time, and the image data is input into the CLIP model with updated parameters for visual defect detection, and the target defect category label of key mine facilities is output. The locally learnable parameters also include local adaptive fusion gating parameters, which are used to dynamically adjust the fusion ratio between the local domain adaptive prompting parameters and the local defect category semantic discriminative prompting parameters. The locally learnable parameters are determined based on historical inspection data or preset values. The CLIP model is configured to extract image features from the image data through an image encoder, generate text description information corresponding to each defect type in the global facility defect category set based on the local learnable parameters, encode the text description information through a text encoder to generate multiple text features, take the text features that match the image features as target text features, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text features. The CLIP model is further configured to extract image features from the image data through an image encoder, and fuse the local adaptive prompting parameters and the local defect category semantic discriminative prompting parameters based on the local adaptive fusion gating parameters to generate fused prompting parameters; The CLIP model is further configured to encode the text description information corresponding to each defect type in the global facility defect category set based on the fusion prompt parameters, and generate multiple text features by encoding the text description information through a text encoder. The CLIP model is further configured to calculate the cosine similarity between image features and each text feature, determine the target text feature that matches the image feature based on the cosine similarity, and output the target defect category label of the key mining facility based on the defect type corresponding to the target text feature.

2. The intelligent visual inspection method for key mining facilities using UAV collaboration as described in claim 1, characterized in that, After updating the extended locally learnable parameters in the locally deployed CLIP model based on the global parameters, the method further includes: The CLIP model is trained based on the total loss function to optimize the locally learnable parameters in the CLIP model. The total loss function includes a contrast loss term, an intra-class aggregation loss term, and an inter-class separation loss term. The contrast loss term is used to align image features with text features. The intra-class aggregation loss term is used to enhance the feature consistency of image features of the same type of defect. The inter-class separation loss term is used to increase the feature differences of different defects. The mathematical expression for the total loss function is: in, Indicates drone client The total loss term of the total loss function, Indicates the comparison loss term, Representing image features, Representing text features, and These represent the weight hyperparameters, Represents the intra-class aggregate loss term. Indicates the first Defect categories in round training The corresponding global category center parameter, Indicates the defect category, Represents the inter-class separation loss term. This represents the set of defect categories that need to be detected locally by the drone. Represents a set of defect categories In terms of defect categories Different defect categories; The intra-class aggregation loss term includes: in, Represents the local dataset of drones The batch of samples is used for gradient descent training. Represents image data, This represents an image encoder based on the CLIP model. This represents the square of the L2 norm, used to calculate the distance between features; The inter-class separation loss term includes: in, This represents preset boundary hyperparameters, including a preset inter-class center distance threshold. This represents the ReLU function, which is used to constrain the loss to occur only when the distance between classes is insufficient.

3. The intelligent visual inspection method for key mining facilities using drone collaboration as described in claim 2, characterized in that, After inputting the image data into the parameter-updated CLIP model for defect visual detection and outputting the target defect category label for key mining facilities, the process further includes: The local category center parameter is calculated based on the visual inspection results of defects during the inspection process, referring to the following formula: in, Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Image encoder representing CLIP model For sample images Extracted image features; The local update amounts of the local adaptive cue parameters and the local defect category semantic discriminative cue parameters are calculated based on the update results of the extended locally learnable parameters in the locally deployed CLIP model, using the following formulas: in, Indicates drone client The local update amount of the local adaptation hint parameter. This indicates the locale adaptability hint parameter. Indicates the first During the training round, the server sends data to the drone client. Global domain adaptability hint parameters, Indicates the corresponding defect category The local update amount of the local defect category semantic discriminative hint parameter. Indicates the first During the training round, the server sends data to the drone client. corresponding defect categories Global defect category semantic discriminative hint parameters, Indicates drone client Corresponding defect category Local defect category semantic discriminative hint parameters; The number of samples corresponding to each local defect category, the local category center parameter, the local update amount of the local domain adaptability prompt parameter, and the local update amount of the local defect category semantic distinguishability prompt parameter are used as upload parameters, and the upload parameters are sent to the server. The server is configured to iteratively aggregate and update the global parameters based on the upload parameters of multiple drone clients.

4. A method for intelligent visual inspection of key mining facilities using unmanned aerial vehicles (UAVs) in collaboration, characterized in that, The method is applied to a server that communicates with each drone in a drone cluster. The server has a CLIP model deployed locally, and the CLIP model includes a text encoder. The method includes: The global defect category semantic discriminative hint parameters corresponding to each defect category of key mining facilities are initialized using a Gaussian distribution; The global domain adaptive prompting parameters are initialized based on the common characteristics of the mine flight environment; Based on the initialized global defect category semantic discriminative hint parameters and global domain adaptive hint parameters, the global facility defect category set is encoded into global category center parameters through template functions and text encoders; The initialized global defect category semantic discriminative hint parameters, global domain adaptive hint parameters, and global category center parameters are used as global parameters, and the global parameters are sent to each UAV in the UAV cluster; Receive the upload parameters sent by the drone, update the global parameters based on the upload parameters, and send the updated global parameters to each drone; The process of receiving upload parameters sent by the drone, updating the global parameters based on the upload parameters, and sending the updated global parameters to each drone includes: Receive the upload parameters sent by the drone, and calculate the domain adaptability aggregation weight of each drone based on the upload parameters, referring to the following formula: in, Indicates drone client Domain-adaptive aggregation weights, Indicates drone The number of inspection samples of all mining facilities in the collected local dataset. Indicates the first The set of participating clients in the round of federated training. Indicates the first The sum of the total number of samples from all clients participating in the training round; The local update amount of the local domain adaptability hint parameter in the uploaded parameters of each UAV is aggregated based on the domain adaptability aggregation weight, and the global domain adaptability hint parameter is updated based on the aggregation result, referring to the following formula: in, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates the first Global domain adaptability hint parameters for the wheel, Indicates drone client The local update amount of the local adaptation hint parameter; The semantic discriminative aggregation weight of each drone is determined based on the uploaded parameters, referring to the following formula: in, Indicates drone client Semantic discriminative aggregation weights, Indicates drone client The local dataset belongs to the defect category The number of samples, This represents the feature consistency weight term. The hyperparameter representing the sensitivity to consistency. Indicates drone client Corresponding category Local category center , and the Wheel category Global category center The squared L2 norm between them is used to represent the characteristic distance between them. The sign indicates a direct proportion; The local update amount of the local defect category semantic discriminative hint parameter in the uploaded parameters of each UAV is aggregated based on the semantic discriminative aggregation weight, and the global defect category semantic discriminative hint parameter is updated based on the aggregation result, as shown in the following formula: in, Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates the first The global defect category semantic discriminative hint parameters for the wheel. Indicates drone client The local update amount of the local defect category semantic discriminative hint parameter; Update the global category center parameter using the following formula: in, It is the momentum coefficient, used to control the degree to which historical center information is retained. Indicates the first The global category center parameter of the wheel, Indicates the first The global category center parameter of the wheel, Indicates the first The local category-weighted average of all participating clients. Indicates drone client Defect categories detected during the inspection process Local category center parameters, Indicates drone client The local dataset belongs to the defect category sample set The number of samples, Indicates participation category Aggregates all drone clients; The updated global category center parameters, global domain adaptability hint parameters, and global defect category semantic discriminative hint parameters are sent to each UAV.

5. The intelligent visual inspection method for key mining facilities using drone collaboration as described in claim 4, characterized in that, After updating the global parameters based on the uploaded parameters and sending the updated global parameters to each drone, the process further includes: Based on the uploaded parameters of each drone, the performance of each drone is evaluated to obtain the cross-domain adaptability index of each drone, referring to the following formula: in, Indicates drone client Cross-domain adaptability indicators Indicates the defect category The corresponding global category center parameter, Indicates drone client The number of defect categories that are detected locally; The learning rate of each UAV is determined based on the aforementioned cross-domain adaptability index, referring to the following formula: in, Indicates the first Wheel of drone client The learning rate Indicates the basic learning rate. Indicates the adjustment factor; The learning rate and the updated global parameters are sent to each UAV. The UAV is configured to use the learning rate as the step size coefficient of the gradient of the local total loss function, and update the local learnable parameters based on the step size coefficient and the updated global parameters.

6. A drone, characterized in that, The drone includes: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor. The drone-coordinated intelligent visual inspection program for key mining facilities is configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described in any one of claims 1 to 3.

7. A server, characterized in that, The server includes: a memory, a processor, and a drone-coordinated intelligent visual inspection program for key mining facilities stored in the memory and capable of running on the processor. The drone-coordinated intelligent visual inspection program for key mining facilities is configured to implement the drone-coordinated intelligent visual inspection method for key mining facilities as described in any one of claims 4 to 5.

8. A drone-assisted intelligent visual inspection system for key mining facilities, characterized in that, The drone-assisted intelligent visual inspection system for key mining facilities includes: multiple drones and a server, with each drone communicating with the server. The drone-assisted intelligent visual inspection system for key mining facilities is configured to implement the drone-assisted intelligent visual inspection method for key mining facilities as described in any one of claims 1 to 3 or 4 to 5.

Images