A smart inspection method and system for collaboration between drones and large-scale models

Abstract

This invention discloses an intelligent inspection method and system based on the collaboration between a UAV and a large-scale model, belonging to the field of UAV technology. It includes: synchronously acquiring multimodal data of the inspection target through a cluster of onboard sensors on the UAV; inputting the synchronized multimodal data into a pre-trained multimodal alignment encoder to generate a unified high-dimensional semantic feature descriptor; inputting the semantic feature descriptor into a large-scale language model integrated with a thought chain technology, triggering the large-scale language model to perform hierarchical progressive causal reasoning and outputting a reasoning conclusion including the fault type, the reasoning logic basis, and a confidence score; when the confidence score is lower than a preset threshold, the large-scale language model autonomously generates flight compensation commands, driving the UAV to perform proactive fine-tuning sampling of the fault area, and iteratively executing the aforementioned steps until the confidence score reaches the target. This invention improves the autonomy, accuracy, and reliability of inspection.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, specifically relating to an intelligent inspection method and system that integrates UAVs and large-scale models. Background Technology

[0002] With the continuous advancement of drone and sensor technologies, their application in intelligent inspection of industrial facilities has become a key technological support for ensuring energy transmission security and stable infrastructure operation. Traditional drone inspection systems mainly rely on preset flight paths to perform tasks, using onboard sensors for image acquisition and storage, which can effectively reduce the risks of manual on-site operations and expand the inspection coverage. However, when facing complex and ever-changing outdoor environments and the increasing demand for refined maintenance, inspection systems face more severe challenges in terms of the depth of environmental perception, the ability to fuse and process multi-source information, and the level of autonomous decision-making in task execution.

[0003] Among these, collaborative inspection solutions that combine large-scale artificial intelligence models have become an important evolutionary path for improving inspection efficiency. This type of method aims to use high-performance algorithms to deeply mine and logically correlate massive amounts of heterogeneous data collected by drones, thereby achieving accurate identification of the target equipment's operating status, fault location, and intelligent diagnosis.

[0004] However, existing technologies largely rely on single-modal image recognition algorithms, which struggle to effectively extract and analyze semantic features of minute hazards such as localized cable overheating or internal insulator damage in extreme environments like fluctuating lighting or cluttered backgrounds, thus limiting detection accuracy. Furthermore, traditional inspection systems generally lack the ability to comprehensively correlate infrared, visible light, and multi-dimensional environmental parameters, hindering deep logical reasoning and causal judgment, leading to frequent false alarms and missed detections. In addition, existing systems typically operate in a unidirectional, passive acquisition mode. When the initial shooting angle is poor or obstruction results in low confidence levels, they cannot dynamically correct the flight platform's pose or actively sample in real time, significantly increasing the workload of subsequent manual re-inspection and severely restricting the closed-loop automation efficiency of inspection tasks. Therefore, a collaborative intelligent inspection solution using drones and large-scale models is desired. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent inspection method and system that coordinates unmanned aerial vehicles (UAVs) and large-scale models, which can effectively solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A smart inspection method that combines drones and large-scale models includes the following specific steps: Step S1: Synchronously acquire multimodal data of the inspection target through the UAV's onboard sensor cluster. The multimodal data includes RGB visible light images, infrared thermal imaging data, and the UAV's own real-time pose parameters. Step S2: Input the synchronized multimodal data into the pre-trained multimodal alignment encoder, and generate a unified high-dimensional semantic feature descriptor through spatial registration and environmental compensation; Step S3: Input the semantic feature descriptor into the large language model integrated with the thinking chain technology, triggering the large language model to perform hierarchical progressive causal reasoning judgment. The hierarchical progressive causal reasoning judgment includes: a first reasoning step, identifying the target structural part based on the semantic feature descriptor; a second reasoning step, associating multimodal abnormal features to identify potential faults; and a third reasoning step, comprehensively judging the potential faults by combining environmental and historical data. In the second reasoning step, the large language model, based on the real-time pose parameters of the UAV and the solar zenith angle and azimuth angle data, solves the specular reflection direction to distinguish between the actual fault temperature rise and the metal surface reflection interference, and outputs a reasoning conclusion including the fault type, reasoning logic basis, and confidence score. Step S4: When the confidence score is lower than the preset threshold, the large language model autonomously generates flight compensation instructions to drive the UAV flight control system to adjust attitude and load parameters, perform active fine sampling of the fault area, and iteratively execute steps S1 to S3 based on the newly collected data until the confidence score reaches or exceeds the preset threshold.

[0007] Furthermore, step S1 further includes: The instantaneous angular velocity ω and linear acceleration a at the moment of acquisition are recorded by an airborne inertial measurement unit, and the point spread function of motion blur is estimated accordingly. The parameters of the point spread function are determined by the angular displacement and linear displacement during the exposure time. Based on the point spread function, the acquired RGB visible light image is deblurred. The deblurred RGB image, infrared thermal imaging data, and the real-time pose parameters of the UAV are timestamped and aligned. The real-time pose parameters are then calculated using multi-source fusion to output a 6-DOF state vector containing longitude, latitude, altitude, pitch angle, roll angle, and yaw angle.

[0008] Furthermore, the deblurring process is performed through a deep convolutional neural network. The input to the network is a three-channel blurred RGB image and a motion parameter map composed of the instantaneous angular velocity ω and linear acceleration a. The output is a restored clear RGB image. The motion parameter map is aligned with the blurred RGB image in the spatial dimension, so that each pixel position is attached with the corresponding ω and a values.

[0009] Furthermore, the spatial registration and environmental compensation in step S2 further include: The homography matrix is ​​calculated using pre-calibrated camera parameters. The temperature values ​​corresponding to the infrared thermal imaging data are superimposed onto the coordinate system of the RGB visible light image with subpixel precision through perspective projection transformation, forming four-channel enhanced image data containing color information and thermal radiation temperature values. The ambient temperature T_env and relative humidity R_H measured by the airborne meteorological station are collected. Combined with the straight-line distance d between the UAV and the inspection target, the thermal radiation temperature value is subjected to environmental compensation correction to obtain the true temperature of the target. The correction is based on the atmospheric transmittance τ, which is determined by the absorption coefficient α, the relative humidity R_H, and the distance d.

[0010] Furthermore, in step S3, the reasoning of the large language model is guided by prompt words and executed layer by layer according to the logical chain of identifying the part, associating anomalies, and making comprehensive judgments. Furthermore, the step of distinguishing between real fault temperature rise and metal surface reflection interference includes: determining whether the bright spot in the infrared image is caused by the specular reflection of sunlight on the metal surface by solving the specular reflection direction; if the angle between the calculated reflected light direction and the camera optical axis is less than a preset threshold, it is determined to be a false alarm interference and the feature point is ignored.

[0011] Furthermore, the confidence score in step S3 is calculated based on the entropy value of the model output probability distribution and the logical consistency between the conclusions of each step in the thought chain reasoning process; The confidence level is high when the reasoning logic of each step is self-consistent and the probability distribution is concentrated; conversely, the confidence level decreases if there is a logical contradiction or the probability distribution is divergent.

[0012] Furthermore, the flight compensation command in step S4 includes at least a movement command and a zoom command; The movement command includes the coordinate offset of the target point in the navigation coordinate system, which is used to drive the drone to move laterally or change altitude to avoid obstructions or obtain a better view. The zoom command includes an adjustment to the onboard optical zoom magnification for closer observation to obtain higher resolution details.

[0013] Furthermore, the active fine-grained sampling in step S4 includes: taking multi-angle coverage shots centered on the fault point, specifically by flying in a circle within a preset radius and taking an image at every preset angle, or by flying close to the fault point at different altitudes.

[0014] Furthermore, if the confidence level still cannot be improved after a preset number of rounds of active sampling, a multi-machine collaborative mode is triggered. In the multi-drone collaborative mode, the large model assigns another nearby drone to take over the task through the cluster scheduling system, and passes the task context, which includes the identified target coordinates, the current logical reasoning state, and the list of features to be verified, to the other drone.

[0015] An intelligent inspection system that coordinates drones and large-scale models includes: A multimodal data acquisition module, configured on the UAV, is used to simultaneously acquire RGB visible light images of the inspection target, infrared thermal imaging data, and the UAV's own real-time pose parameters. The multimodal alignment encoder module is used to generate a unified high-dimensional semantic feature descriptor by spatial registration and environmental compensation of the synchronized multimodal data; The large-scale reasoning module of the Thinking Chain integrates a large language model to receive the semantic feature descriptors and perform hierarchical and progressive causal reasoning judgments. The causal reasoning judgments include: identifying target structural parts, associating multimodal abnormal features, and combining environmental and historical data for comprehensive analysis. The large-scale reasoning module of the Thinking Chain is also configured to distinguish between real fault temperature rise and metal surface reflection interference by solving the specular reflection direction based on the real pose parameters of the UAV and the solar zenith angle and azimuth angle data, and output a reasoning conclusion including fault type, reasoning logic basis, and confidence score. The closed-loop control and active sampling module is used to generate flight compensation commands autonomously by the large language model when the confidence score is lower than a preset threshold, drive the UAV flight control system to adjust attitude and load parameters, and perform active fine sampling of the fault area.

[0016] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention breaks through the technical bottleneck of traditional inspection relying on single visible light recognition. By simultaneously acquiring red, green, and blue images, infrared thermal imaging, and high-precision pose data, it achieves full-spectrum perception of the inspection target. Utilizing a multimodal alignment encoder to map heterogeneous data to a unified feature space not only solves the spatiotemporal consistency problem between different sensors but also significantly improves the extraction accuracy of minute fault features in extreme environments such as fluctuating lighting and complex backgrounds. This results in a significant improvement in the accuracy of identifying hidden defects such as internal insulator damage and localized cable overheating.

[0017] 2. Unlike traditional image classification algorithms that rely solely on statistical probability, this invention combines physical laws, historical equipment status, and multi-dimensional environmental parameters for comprehensive analysis. By systematically eliminating interfering factors through a logical chain, such as accurately distinguishing between metal reflective hotspots and actual fault temperature rises, it significantly reduces false alarm and missed alarm rates during inspections. The introduction of the logical chain enables the system to provide interpretable diagnostic results, offering clear decision-making support for subsequent manual maintenance.

[0018] 3. This invention constructs a complete closed-loop control system from perception to reasoning to action, completely changing the traditional passive data collection operation mode of UAVs. By using a large model to evaluate the confidence level of reasoning in real time and autonomously generate pose correction commands, the system can drive the UAV to perform real-time re-shooting and close-up observation when there is uncertainty in the judgment. This active sampling mechanism greatly reduces the need for manual secondary inspection due to poor quality of raw data, significantly improves the efficiency of the automated closed-loop of inspection tasks, significantly reduces inspection costs, and shortens the response cycle.

[0019] 4. This invention can be quickly adapted to various industrial inspection scenarios. Real-time compensation and correction of environmental parameters ensures accurate temperature measurement under different altitudes, humidity levels, and weather conditions. Furthermore, the semantic feature-based description method allows the system to accommodate more types of sensor inputs, laying a solid technical foundation for the continuous upgrading of future inspection equipment and multi-machine collaborative operations. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution for an intelligent inspection method that combines drones and large-scale models. Figure 2 This is a schematic diagram illustrating the core principle of causal reasoning and judgment based on the large-scale thinking chain model; Figure 3 It is a logical flowchart of semantic alignment and multidimensional feature fusion for multimodal heterogeneous data; Figure 4 This is a schematic diagram of the multi-level interaction and data flow between the inference confidence-driven closed-loop control of the UAV and the active sampling module. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1-4 The present invention will be further described in detail below with reference to specific embodiments.

[0022] The first aspect is the intelligent inspection method for collaboration between drones and large models disclosed in this application, which is implemented according to the following steps: The first step, S1, involves simultaneously acquiring RGB visible light images of the inspected target, long-wave infrared thermal imaging data, and the UAV's real-time pose parameters using the UAV's onboard sensor cluster, ensuring that these three types of data are strictly aligned in time. Synchronous and high-quality multimodal data forms the basis for subsequent heterogeneous feature fusion and causal inference. To achieve this, this step is further subdivided into the following sub-steps.

[0023] Step S101: Configure the acquisition parameters of the airborne sensor; set the resolution of the RGB camera on the UAV to 3840×2160 or higher, and the sampling frame rate to no less than 30 frames / second, so as to ensure that clear texture and geometric structure information of the surface of the inspection target can be extracted during dynamic flight.

[0024] The infrared acquisition unit employs an uncooled focal plane array detector, operating in the 8μm to 14μm band with a temperature resolution better than 0.05℃, used for real-time monitoring of target surface temperature distribution and capturing abnormal hotspots. The UAV's attitude parameters are calculated through multi-source fusion using an onboard inertial measurement unit, a global navigation and positioning system, and a barometric altimeter, outputting a 6-DOF state vector including longitude, latitude, altitude, pitch angle, roll angle, and yaw angle.

[0025] Step S102: Establish a high-precision timestamp synchronization mechanism. The onboard computer uses a precise time protocol or satellite timing signal to send synchronization pulses to the visible light camera, infrared thermal imager, and inertial measurement unit. Each frame of RGB and infrared images is time-stamped with nanosecond precision at the moment of acquisition and correlated with the UAV's pose parameters at the current moment. The time synchronization error of the entire system is controlled within 10 milliseconds.

[0026] Step S103: Perform motion deblurring on images under dynamic conditions. When the drone is cruising at high speed or encountering turbulent airflow, the acquired images may exhibit motion blur. The system uses an inertial measurement unit to record the instantaneous angular velocity at the moment of capture. and linear acceleration Based on this, the point spread function of motion blur is estimated. The parameters of the point spread function are determined by the angular displacement during the exposure time. and linear displacement The decision, and the specific calculation method, are as follows: in For camera exposure time, It is an angular displacement. This is linear displacement. [The following is a possible interpretation:] and Projecting the image onto the image plane yields the shape and size of the blur kernel.

[0027] The system provides two optional deblurring algorithms: Method 1: Wiener filtering based on point spread function. Based on the calculated blur kernel, Wiener filtering is performed in the frequency domain to recover the clear image. The signal-to-noise ratio parameter is adaptively set according to the local variance of the image.

[0028] Method 2: Deep Convolutional Neural Network Deblurring. This network adopts an encoder-decoder structure, with the encoder consisting of 5 convolutional layers with a stride of 2, and the decoder consisting of 5 upsampling convolutional layers. Skip connections are added between the encoder and decoder.

[0029] The network input consists of a three-channel blurred RGB image and instantaneous angular velocity recorded by the inertial measurement unit. and linear acceleration The resulting 2-channel motion parameter map is spatially aligned with the image, meaning that a corresponding parameter is appended to each pixel location. and Value. The dimension of the input tensor is... ,in and These represent the image height and width, respectively. The network output is the restored, clear RGB image with dimensions of [dimensions missing]. .

[0030] The training process of this deep convolutional neural network is as follows: Construction of training data: A large number of clear UAV inspection images were collected as ground truth references. Simulation software was used to train the data based on the measured UAV angular velocity. and linear acceleration A blur kernel is randomly generated based on the probability distribution. The sharp image is convolved with this blur kernel, and Gaussian noise is added to generate the corresponding motion-blurred image. Each pair of sharp and blurred images constitutes a training sample.

[0031] In addition, blurry images with inertial data were collected during actual flight and registered with clear reference frames taken by a high-precision gimbal in the same scene as a supplementary training set to enhance the model's adaptability to the real environment.

[0032] Loss function design: A composite loss function is used. Content loss The L1 distance between the deblurred result and the sharp image is... Perceived loss Using a pre-trained VGG-16 network, feature maps of the deblurred and sharpened images at the conv3_3 layer are extracted, and their mean squared error is calculated. Adversarial loss is then applied. It is implemented through a discriminator network, which consists of 5 convolutional layers and 2 fully connected layers. The input is an image, and the output is a discrimination score between 0 and 1.

[0033] The deblurring network is used as a generator, and the discriminator and generator are trained alternately: the discriminator is trained to distinguish between real clear images and deblurred images output by the generator, and its loss function is binary cross-entropy; the generator is trained to minimize the probability that the discriminator judges it as false, that is, to maximize the adversarial loss. and The preset weighting coefficients are set to 0.1 and 0.01 respectively.

[0034] Training objective: To enable the neural network to accurately recover clear image texture and edges from a blurred input image and corresponding inertial motion parameters, thus ensuring that visual features remain clearly discernible even under severe shaking. The training convergence condition is the total loss... It no longer decreases for 10 consecutive cycles on the validation set.

[0035] Step S104: Associate pose parameters with image frames; each frame of the image after motion deblurring is associated with the synchronously acquired pose parameters. The association method is as follows: the pose parameters are encoded into a vector of length 10, including longitude, latitude, altitude, pitch angle, roll angle, yaw angle, instantaneous northward velocity, instantaneous eastward velocity, instantaneous vertical velocity, and timestamp.

[0036] The vector and the corresponding image pixel data are encapsulated together as a structured data record and stored in the onboard computer's memory queue. The image data is stored in JPEG compression format, and the pose vector is stored in binary floating-point format; both are bound together using the same frame number index.

[0037] Step S105: Output synchronized and aligned multimodal data streams. After the above processing, the system outputs a set of strictly time-aligned multimodal data streams: each data unit contains an RGB image, an infrared thermal image, and the corresponding UAV pose parameter vector. All data have a unified timestamp identifier, and motion blur of the images has been effectively suppressed. This data stream is pushed to the next processing module in real time through the airborne communication interface.

[0038] In summary, step S1 synchronously acquires RGB images, infrared thermal images, and UAV pose parameters through airborne sensors, eliminates the impact of jitter by using timestamp synchronization and motion deblurring algorithms, and finally outputs a time-aligned and quality-enhanced multimodal data stream, providing input for subsequent feature fusion and inference judgment.

[0039] The next step, S2, involves receiving the multimodal data stream output from step S1, namely, time-aligned RGB images, infrared thermal images, and UAV pose parameter vectors. These data come from different sensors, with varying spatial coordinate systems and physical dimensions. The goal of this step is to map them to a unified high-dimensional semantic feature space, eliminate geometric biases, and combine them with environmental parameters to generate feature descriptors that can be directly understood by large models. This is specifically divided into the following sub-steps.

[0040] Step S201: Load a pre-trained multimodal alignment encoder; the system loads a pre-trained multimodal alignment encoder. This encoder adopts a deep neural network architecture based on contrastive learning, which contains three parallel processing branches: an image feature extraction branch, a temperature field feature processing branch, and a pose space geometric transformation branch.

[0041] The image feature extraction branch processes RGB images, employing either a ResNet-50 residual network or a ViT-Base visual transformer as the backbone network to extract multi-scale texture and topological features. This branch outputs a 2048-dimensional feature vector.

[0042] The temperature field feature processing branch is used to process infrared thermal images. It employs a lightweight convolutional neural network containing four convolutional layers and two pooling layers, specifically designed to extract features such as temperature gradients, hotspot distribution, and isotherm shapes. This branch outputs a 512-dimensional feature vector.

[0043] The pose space geometric transformation branch is used to process the UAV pose parameters, namely the six values ​​of longitude, latitude, altitude, pitch angle, roll angle, and yaw angle. These parameters are mapped into a 256-dimensional geometric feature vector through three fully connected layers.

[0044] This multimodal alignment encoder needs to be pre-trained. The training process is as follows: Training data construction: A large amount of labeled UAV inspection data was collected. Each data set includes an RGB image, an infrared thermal image, pose parameters, and a text description label for the same scene. The text description label indicates the target type and status in the scene, such as "insulator string normal" or "cable joint overheating". Different modal data from the same scene are used as positive sample pairs, and arbitrary modal data from different scenes are used as negative sample pairs. Specifically, for each pair of modes A and B, the positive sample pair comes from the same target at the same time, and the negative sample pair comes from different times or different targets.

[0045] Loss function design: A contrastive loss function, InfNCE, is used. For each batch, the cosine similarity of the feature vectors of positive sample pairs and the similarity of negative sample pairs are calculated. The loss function takes the form of... in and The first Feature vectors output by two different modal branches in a sample. The cosine similarity function is used. The temperature coefficient is set to 0.07. denoted as the total number of samples in the batch. This loss function encourages different modalities from the same target to be close together in space, while features from different targets are kept far apart.

[0046] Training objective: To enable the three branches to learn cross-modal semantic alignment capability, meaning that regardless of whether the input is an RGB image, infrared image, or pose parameters, as long as it describes the same state of the same inspected target, their mapped feature vectors should be highly similar in high-dimensional space. The training convergence condition is the loss value... It is less than 0.1 on the validation set and does not decrease for 5 consecutive periods.

[0047] Step S202: Perform spatial registration and image correction. Using the pre-calibrated relative position parameters, intrinsic parameter matrices, and focal length information of the visible light and infrared cameras, calculate the homography matrix between the two camera coordinate systems. Through perspective projection transformation, the temperature values ​​from the infrared thermal imaging data are superimposed onto the corresponding coordinate system of the visible light image with sub-pixel precision. Specifically, for each pixel coordinate (u, v) in the visible light image, the corresponding sub-pixel position (u', v') in the image corresponding to the infrared thermal imaging data is found based on the homography matrix H, and bilinear interpolation is used to obtain the thermal radiation temperature value at that position.

[0048] After the overlay is completed, each pixel not only contains RGB three-channel color information, but also has an additional thermal radiation temperature value, forming four-channel enhanced image data, denoted as RGBA_T, where A_T represents the temperature channel.

[0049] Step S203: Collect environmental parameters and perform temperature compensation; to eliminate the influence of environmental factors on the accuracy of infrared thermometry, the UAV-borne micro weather station collects the following two key environmental parameters in real time: atmospheric temperature. and relative humidity of air These two parameters are input into the environmental compensation correction model. This model, based on Planck's law and atmospheric transmittance theory, applies the apparent temperature measured by the infrared sensor. Make corrections to obtain the target true temperature. The compensation formula is as follows: The meanings of each letter are as follows: The target true temperature obtained after environmental compensation, in degrees Celsius; Apparent temperature directly measured by an infrared sensor, in degrees Celsius; Emissivity of the target surface, ranging from 0 to 1, with different preset values ​​for different materials, such as 0.85 for alumina and 0.25 for galvanized steel; Atmospheric ambient temperature measured by an airborne weather station, in degrees Celsius.

[0050] Atmospheric transmittance, dimensionless, is calculated using the following empirical formula: . Absorption coefficient, a fixed value of 0.02, in units of percentage per meter; : Relative humidity of air, expressed as a percentage, ranging from 0 to 100; The straight-line distance between the UAV and the inspection target, in meters, is calculated from the coordinates in the UAV's pose parameters and the target's preset coordinates.

[0051] Step S204, Subpixel-level feature registration and semantic vector correction: During semantic alignment, the system performs subpixel-level spatial registration of the rust texture features in the RGB image and the abnormal temperature rise features in the infrared image. By calculating the feature similarity score, the environmental humidity data is incorporated into the correction coefficient of the semantic vector to eliminate the attenuation error caused by the absorption of infrared radiation by water mist or atmospheric aerosols.

[0052] The specific method is as follows: On the RGBA_T image generated in step S202, key points are extracted using a scale-invariant feature transform algorithm. Then, sub-pixel-level tracking of feature points between adjacent frames is performed using optical flow to ensure that features of different modalities are aligned in the same spatial location. Afterwards, the ambient humidity... As a correction factor, multiply by the value of the infrared temperature channel to obtain the humidity-compensated temperature value.

[0053] Step S205: Generate a multi-dimensional semantic feature descriptor. After the above processing, the multimodal alignment encoder receives the four-channel enhanced image after spatial registration and environmental compensation, as well as the original pose parameters, and outputs a 512-dimensional or 1024-dimensional dense semantic feature descriptor. This descriptor is a floating-point vector that not only encodes the visual shape information of the target, such as shape, color, and texture, but also embeds physical state attributes, such as apparent thermal resistance, emissivity deviation, and relative temperature difference. Each dimension of the descriptor does not have a direct physical meaning, but its overall position in high-dimensional space represents the comprehensive semantic state of the target.

[0054] After being trained through contrastive learning, the multimodal aligned encoder can semantically equivalence heterogeneous signals. Image signals and sensor values ​​that originally had different properties are uniformly mapped to the same feature space, enabling subsequent large model logic layers to understand this visual and physical information as if they were text.

[0055] In summary, step S2 completes the transformation from raw sensor data to semantic feature vectors. The core work includes: achieving cross-modal semantic unification using a multimodal alignment encoder trained through contrastive learning; achieving sub-pixel-level spatial registration of infrared and visible light through homography transformation; performing environmental compensation for infrared thermometry using atmospheric temperature and relative humidity collected by an airborne weather station, where the meaning of each letter in the compensation formula is clearly defined; and finally generating dense semantic descriptors suitable for large-scale model inference. These descriptors will serve as input to step S3 for causal inference judgment based on the thought chain-based large-scale model.

[0056] The next step, S3, involves receiving the semantic feature descriptor generated in step S2 and inputting it into a large language model integrated with thought chain technology. This model, combined with a pre-defined domain knowledge graph, performs a progressive logical deduction of the physical state of the inspection target, identifies potential fault hazards, and determines their causes. Specifically, this involves the following sub-steps.

[0057] Step S301: Deploy and load the large language model; the large language model is deployed on a ground workstation or an edge server equipped with high-performance computing units. The model has at least 13 billion parameters to ensure sufficient inference capabilities. The model has been deeply fine-tuned on specialized corpora in specific industrial fields such as power systems, rail transportation, or petrochemicals, enabling it to have prior knowledge of typical equipment faults such as insulator spontaneous explosion, cable strand breakage, and bolt loosening.

[0058] The fine-tuning process of this large language model is as follows: Construction of fine-tuned data: A large amount of text-label pair data from industrial inspection scenarios was collected. Each data point contains a natural language description of the equipment status, such as "obvious cracks appear on the surface of the insulator string, and the infrared image shows that the local temperature rise exceeds the ambient temperature by 15 degrees Celsius," and a corresponding fault type label, such as "early signs of insulator spontaneous explosion." The ratio of positive to negative samples in the training dataset is approximately 1:3, with negative samples including normal conditions and common interference situations, such as false hotspots caused by reflections on the metal surface.

[0059] Loss function design: The cross-entropy loss function is adopted, in the form of... in This represents the total number of fault categories. One-hot encoding of the real label. The model predicts the first The probability of a class.

[0060] Training objective: To enable the large language model to accurately output the corresponding fault type and confidence level based on the input text description. The training convergence condition is that the classification accuracy on the validation set reaches above 95%, and the cross-entropy loss value decreases by no more than 0.01 for three consecutive epochs.

[0061] After fine-tuning, the model possesses the ability to identify faults. During actual inference, the model receives the semantic feature descriptors generated in step S2. A linear mapping layer transforms the 512-dimensional or 1024-dimensional semantic descriptors into vectors with the same word embedding dimensions as the model, which are then used as the model's input prefix.

[0062] Step S302: Input semantic feature descriptors and initiate thought chain reasoning; feed the semantic feature descriptors into the loaded large language model after linear mapping. The model integrates thought chain technology, which requires the model to explicitly generate intermediate reasoning steps before outputting the final decision. The system guides the model to reason in a hierarchical and progressive manner through prompt words.

[0063] The specific execution process of the thought chain reasoning is as follows: Step 1: Identify the target structural component. Based on the visual shape and pose information in the semantic feature descriptor, the model determines the specific type and location of the currently observed object, such as identifying it as a drain line pressure pipe of a power tower.

[0064] Step 2: Correlate multimodal anomaly features. The model simultaneously analyzes texture anomalies under visible light, such as the presence of black oxides, deformation, or cracks on the surface, and temperature rise anomalies under infrared light. Step S2 has already performed environmental compensation for the infrared thermometry, outputting the true temperature value. The model directly reads this temperature value and compares it with a preset temperature difference threshold. If the difference between the true temperature and the ambient temperature exceeds 15 degrees Celsius, a temperature rise anomaly is determined to exist. If both modes indicate anomalies, proceed to the next step.

[0065] Step 3: Conduct a comprehensive analysis by combining environmental and historical data. The model uses current load current, ambient temperature, humidity, wind speed, and historical inspection records to determine whether the anomaly conforms to normal physical loss patterns or environmental heat reflection patterns. For example, in sunny weather, metal surfaces may experience temperature rise due to sunlight, but this is usually uniformly distributed and unrelated to the load current; while fault temperature rise is often concentrated at connection points and varies with the load.

[0066] Step S303: Perform false hotspot detection. During the logical deduction process, the model will conduct multi-dimensional checks on the bright spots identified in the infrared image to distinguish between real faults and false interference. The specific method is as follows: The system utilizes a large model combined with the UAV's current flight attitude parameters, including pitch, roll, yaw angles, as well as solar zenith and azimuth angles, to calculate the reflective geometry of the target metal surface. By determining the direction of specular reflection, it judges whether the hotspot is caused by specular reflection of sunlight on the metal surface. If the angle between the calculated reflected ray direction and the camera's optical axis is less than a preset threshold of 15 degrees, it is determined to be a false alarm interference, and the model automatically ignores the feature point.

[0067] For areas suspected of being real hotspots, the model directly uses the actual temperature value obtained in step S2 after compensation. This actual temperature is compared with historical normal values ​​and the average value of similar equipment. If the deviation exceeds a preset threshold and cannot be explained by environmental factors, it is determined to be a real fault.

[0068] Step S304: Output the reasoning conclusion and confidence level; after completing the above reasoning, the large language model outputs a structured judgment result, including the following: The text describes the type of fault, such as poor contact of the drain line connector leading to electrochemical corrosion.

[0069] The spatial coordinates of the fault location are calculated from the UAV pose parameters and image coordinates.

[0070] The logical basis of reasoning, that is, the intermediate conclusions of each step in the chain of thought.

[0071] A confidence score between 0 and 1. The confidence score is calculated based on the entropy of the model's output probability distribution and the consistency of inference results across multiple rounds. Let the class probability distribution of the model's output be... Then the entropy value .

[0072] Simultaneously, check the logical consistency between the conclusions of each step in the thought process; deduct points if contradictions are found. Final confidence level. ,in The consistency coefficient is 1 when there is no contradiction and 0.5 when there is a contradiction. The confidence level is high when the reasoning steps are logically consistent and the probability distribution is concentrated; conversely, the confidence level decreases if there is a contradiction or the distribution is divergent.

[0073] In summary, this step utilizes a large language model integrating thought chain technology to perform causal reasoning on the semantic feature descriptors generated in step S2. Through hierarchical and progressive reasoning logic, combined with geometric reflection analysis and the real temperature compensated for in step S2, it effectively distinguishes between real faults and spurious interference, and outputs interpretable diagnostic results. The fault determination and confidence level output in step S3 will serve as the triggering basis for the closed-loop interactive feedback and active sampling control in step S4. When the confidence level is lower than the threshold, the system will activate the active sampling mechanism.

[0074] Finally, in step S4, the inference conclusion and confidence score output from step S3 are received. The system dynamically evaluates the current sampling quality based on the confidence score. When the confidence score falls below a preset threshold, the large model autonomously generates flight compensation commands, driving the UAV flight control system to adjust attitude and load parameters, achieving proactive and refined sampling of the fault area. This step connects perception, inference, and action into a closed-loop control system. Specifically, it consists of the following sub-steps.

[0075] Step S401: Evaluate the inference confidence level and compare it with a threshold; the system reads the confidence score output in step S304. The score, ranging from 0 to 1, was calculated in step S3 based on the entropy value of the model's output probability distribution and the logical consistency of the thought chain. The system presets a confidence threshold, for example, 0.75.

[0076] like This indicates that the current reasoning result is reliable, and no additional sampling is needed; we can proceed directly to the conclusion generation stage.

[0077] like If the current data quality is insufficient to support a reliable judgment, the active sampling process will be triggered.

[0078] Typical reasons for triggering sampling include: severe occlusion at the shooting angle, insufficient lighting contrast, excessive shooting distance leading to incomplete feature extraction, or logical contradictions in the thought process of step S3. In these cases, the probability distribution output by step S3 tends to be discrete, the entropy value increases, and the confidence level decreases accordingly.

[0079] In step S402, the large model generates flight compensation commands. If the confidence level is insufficient, the large model, as the decision-making core, generates specific flight compensation commands based on the current task context. The large model internally stores the context state of the current inspection task, including the coordinates of the identified targets, the pose parameters of the current UAV, and the specific reasons for the uncertainty in step S3, such as whether the visible light texture is blurred or the infrared temperature rise data is suspicious.

[0080] Flight compensation instructions are divided into two categories: Movement command: Includes the target point's coordinate offset in the Northeast-Eastern Sky navigation coordinate system. Delta Y The units are meters. These offsets are calculated based on the difference between the current shooting angle and the ideal angle. For example, if the target is obstructed, the large model can instruct the drone to move 5 meters laterally to avoid the obstruction.

[0081] Zoom commands: These include adjustments to the onboard optical zoom ratio, expressed in magnification. For example, if the shooting distance is too far and details are unclear, the large model can command the zoom ratio to increase from the current 2x to 5x.

[0082] Step S403: The flight control system executes commands. After receiving the movement and zoom commands, the UAV flight control system executes the proportional-integral-derivative (PID) control algorithm. The PID controller calculates the thrust adjustment required by the power system based on the deviation between the current position and the target position, driving the UAV smoothly to the designated coordinate point.

[0083] Simultaneously, the gimbal motor adjusts the lens focal length according to zoom commands. The response time of the entire control loop is less than 500 milliseconds, ensuring real-time performance.

[0084] Step S404: Perform multi-angle coverage photography. After the drone reaches the new location, it does not only take a single image, but performs multi-angle coverage photography of the suspected fault area. Specifically, it flies in a circle with a radius of 5 meters around the fault point, taking an image every 45 degrees; or it flies close to the fault area at different altitudes, such as 2 meters higher and 2 meters lower than the original altitude.

[0085] During this process, the drone transmits newly acquired high-definition images and infrared thermal imaging data back in real time. These data are then synchronized in time in step S1 and feature extracted in step S2 to generate new semantic feature descriptors, which are then input into the large model in step S3 for re-inference.

[0086] Step S405: Iterate until the confidence level reaches the target or multi-machine collaboration is triggered; the system repeats steps S401 to S404, recalculating the confidence level after each round of sampling. Once the confidence level reaches or exceeds the preset threshold of 0.75, active sampling stops and the system proceeds to the conclusion generation stage.

[0087] If the confidence level still cannot be improved after three consecutive rounds of sampling, or if the remaining battery power of the drone is lower than a safety threshold, such as 20%, then the multi-drone collaborative mode will be triggered.

[0088] In multi-drone collaborative mode, the large model assigns another nearby drone with the same payload capacity to take over the task through the cluster scheduling system.

[0089] During takeover, the system needs to transfer the complete task context, including: the identified target coordinates, the current logical reasoning state (e.g., excluded interference types), and a list of features to be verified (e.g., temperature rise points to be confirmed). The receiving drone takes off from its current location, flies directly to the target area, and continues sampling using its sensors. The large model seamlessly transfers the completed reasoning chain to the new drone, ensuring logical continuity.

[0090] In summary, step S4 achieves closed-loop control from reasoning to action. When confidence is insufficient, the large model autonomously generates pose adjustment and zoom commands, driving the UAV to perform refined, multi-angle active sampling. If a single UAV cannot complete the task, the system supports multi-UAV collaborative takeover. Through this closed-loop mechanism, the system can proactively supplement missing information, significantly improving the automation level and reliability of inspections. After step S4 is completed, the system will enter the conclusion generation stage and output the final inspection report.

[0091] Finally, this method also involves a conclusion generation stage. The final inspection report is not a simple image summary, but includes a textual description of the fault type, the spatial coordinates of the fault location, detailed logical justification, and suggested repair solutions. Specifically, the temperature rise difference is derived from a comparison between the actual temperature after compensation in step S203 and the historical normal value; the oxide layer area ratio is calculated using the semantic segmentation results of the image feature extraction branch in step S201; and the judgment logic for excluding reflection interference directly references the reasoning conclusion from step S303.

[0092] In power tower inspection scenarios, the system can simultaneously monitor insulators, crimped pipes, drain wires, and fastening bolts. The report details conclusions such as "electrochemical corrosion caused by poor bolt contact was detected," along with temperature rise differences, oxide layer area ratios, and judgment logic to exclude reflected interference. The large model interacts with the drone in real time via a low-latency communication link, such as 5G or a dedicated microwave link. The response latency for a single inference and command generation is less than a preset latency threshold of less than 500 milliseconds, ensuring the real-time performance of the inspection task.

[0093] On the other hand, the intelligent inspection system for collaborative operation of drones and large models disclosed in this application includes: A multimodal data acquisition module, configured on the UAV, is used to simultaneously acquire RGB visible light images of the inspection target, infrared thermal imaging data, and the UAV's own real-time pose parameters. The multimodal alignment encoder module is used to generate a unified high-dimensional semantic feature descriptor by spatial registration and environmental compensation of synchronized multimodal data; The large-scale reasoning module of the Thinking Chain integrates a large language model to receive semantic feature descriptors and perform hierarchical and progressive causal reasoning judgments. The causal reasoning judgments include: identifying target structural parts, associating multimodal abnormal features, and making comprehensive judgments by combining environmental and historical data. The large-scale reasoning module of the Thinking Chain is also configured to distinguish between real fault temperature rise and metal surface reflection interference by solving the mirror reflection direction based on the real pose parameters of the UAV and the solar zenith angle and azimuth angle data, and output reasoning conclusions including fault type, reasoning logic basis and confidence score. The closed-loop control and active sampling module is used to generate flight compensation commands autonomously by the large language model when the confidence score is lower than the preset threshold, drive the UAV flight control system to adjust the attitude and load parameters, and perform active fine sampling of the fault area.

[0094] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0095] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An intelligent inspection method of a UAV cooperating with a large model, characterized in that, Includes the following steps: Step S1: Synchronously acquire multimodal data of the inspection target through the UAV's onboard sensor cluster. The multimodal data includes RGB visible light images, infrared thermal imaging data, and the UAV's own real-time pose parameters. Step S2: Input the synchronized multimodal data into the pre-trained multimodal alignment encoder, and generate a unified high-dimensional semantic feature descriptor through spatial registration and environmental compensation; Step S3: Input the semantic feature descriptor into the large language model integrated with the thinking chain technology, triggering the large language model to perform hierarchical progressive causal reasoning judgment. The hierarchical progressive causal reasoning judgment includes: a first reasoning step, identifying the target structural part based on the semantic feature descriptor; a second reasoning step, associating multimodal abnormal features to identify potential faults; and a third reasoning step, comprehensively judging the potential faults by combining environmental and historical data. In the second reasoning step, the large language model, based on the real-time pose parameters of the UAV and the solar zenith angle and azimuth angle data, solves the specular reflection direction to distinguish between the actual fault temperature rise and the metal surface reflection interference, and outputs a reasoning conclusion including the fault type, reasoning logic basis, and confidence score. Step S4: When the confidence score is lower than the preset threshold, the large language model autonomously generates flight compensation instructions to drive the UAV flight control system to adjust attitude and load parameters, perform active fine sampling of the fault area, and iteratively execute steps S1 to S3 based on the newly collected data until the confidence score reaches or exceeds the preset threshold.

2. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, Step S1 further includes: The instantaneous angular velocity ω and linear acceleration a at the moment of acquisition are recorded by an airborne inertial measurement unit, and the point spread function of motion blur is estimated accordingly. The parameters of the point spread function are determined by the angular displacement and linear displacement during the exposure time. Based on the point spread function, the acquired RGB visible light image is deblurred. The deblurred RGB image, infrared thermal imaging data, and the real-time pose parameters of the UAV are timestamped and aligned. The real-time pose parameters are then calculated using multi-source fusion to output a 6-DOF state vector containing longitude, latitude, altitude, pitch angle, roll angle, and yaw angle.

3. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 2, characterized in that, The deblurring process is performed by a deep convolutional neural network. The input to the network is a three-channel blurred RGB image and a motion parameter map consisting of the instantaneous angular velocity ω and linear acceleration a. The output is a restored clear RGB image. The motion parameter map is aligned with the blurred RGB image in the spatial dimension, so that each pixel position is assigned a corresponding ω and a value.

4. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, The spatial registration and environmental compensation in step S2 further include: The homography matrix is ​​calculated using pre-calibrated camera parameters. The temperature values ​​corresponding to the infrared thermal imaging data are superimposed onto the coordinate system of the RGB visible light image with subpixel precision through perspective projection transformation, forming four-channel enhanced image data containing color information and thermal radiation temperature values. The ambient temperature T_env and relative humidity R_H measured by the airborne meteorological station are collected. Combined with the straight-line distance d between the UAV and the inspection target, the thermal radiation temperature value is subjected to environmental compensation correction to obtain the true temperature of the target. The correction is based on the atmospheric transmittance τ, which is determined by the absorption coefficient α, the relative humidity R_H, and the distance d.

5. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, The reasoning of the large language model in step S3 is guided by prompt words and is executed layer by layer according to the logical chain of identifying the part, associating anomalies, and making comprehensive judgments. Furthermore, the step of distinguishing between real fault temperature rise and metal surface reflection interference includes: determining whether the bright spot in the infrared image is caused by the specular reflection of sunlight on the metal surface by solving the specular reflection direction; if the angle between the calculated reflected light direction and the camera optical axis is less than a preset threshold, it is determined to be a false alarm interference and the feature point is ignored.

6. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, The confidence score in step S3 is calculated based on the entropy value of the model output probability distribution and the logical consistency between the conclusions of each step in the thought chain reasoning process; The confidence level is high when the reasoning logic of each step is self-consistent and the probability distribution is concentrated; conversely, the confidence level decreases if there is a logical contradiction or the probability distribution is divergent.

7. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, The flight compensation commands in step S4 include at least movement commands and zoom commands; The movement command includes the coordinate offset of the target point in the navigation coordinate system, which is used to drive the drone to move laterally or change altitude to avoid obstructions or obtain a better view. The zoom command includes an adjustment to the onboard optical zoom magnification for closer observation to obtain higher resolution details.

8. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, The active fine-grained sampling in step S4 includes: taking multi-angle coverage shots centered on the fault point, specifically by flying in a circle within a preset radius and taking an image at every preset angle, or by flying close to the fault point at different altitudes.

9. The intelligent inspection method for collaborative operation of UAVs and large-scale models according to claim 1, characterized in that, If the confidence level still cannot be improved after a preset number of rounds of active sampling, the multi-machine collaborative mode will be triggered. In the multi-drone collaborative mode, the large model assigns another nearby drone to take over the task through the cluster scheduling system, and passes the task context, which includes the identified target coordinates, the current logical reasoning state, and the list of features to be verified, to the other drone.

10. An intelligent inspection system for collaborative operation of unmanned aerial vehicles (UAVs) and large-scale models, used to perform the method as described in any one of claims 1 to 9, characterized in that, include: A multimodal data acquisition module, configured on the UAV, is used to simultaneously acquire RGB visible light images of the inspection target, infrared thermal imaging data, and the UAV's own real-time pose parameters. The multimodal alignment encoder module is used to generate a unified high-dimensional semantic feature descriptor by spatial registration and environmental compensation of the synchronized multimodal data; The large-scale reasoning module of the Thinking Chain integrates a large language model to receive the semantic feature descriptors and perform hierarchical and progressive causal reasoning judgments. The causal reasoning judgments include: identifying target structural parts, associating multimodal abnormal features, and combining environmental and historical data for comprehensive analysis. The large-scale reasoning module of the Thinking Chain is also configured to distinguish between real fault temperature rise and metal surface reflection interference by solving the specular reflection direction based on the real pose parameters of the UAV and the solar zenith angle and azimuth angle data, and output a reasoning conclusion including fault type, reasoning logic basis, and confidence score. The closed-loop control and active sampling module is used to generate flight compensation commands autonomously by the large language model when the confidence score is lower than a preset threshold, drive the UAV flight control system to adjust attitude and load parameters, and perform active fine sampling of the fault area.

Images