Unmanned aerial vehicle autonomous flight inspection method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of industrial equipment inspection, in particular to an unmanned aerial vehicle autonomous flight inspection method and device, electronic equipment and a storage medium. Splicing the original visible light images at different visual angles to obtain a panoramic visible light image; constructing a quantization energy field according to the infrared thermal imaging image and the ultraviolet imaging image, and processing the panoramic visible light image by using the quantization energy field to obtain a first fusion image; performing gradient processing on the infrared thermal imaging image and the ultraviolet imaging image, and adjusting the pixel value of the first fusion image by using gradient information to obtain a second fusion image; and performing color cast compensation on the second fusion image to obtain a multi-modal target image. Compared with an original visible light image, the image quality of the multi-modal target image obtained through processing is greatly improved, details of the industrial equipment in the target environment can be reflected more clearly, the accuracy of judgment of the fault type and degree of the industrial equipment is remarkably improved, and the inspection efficiency of the unmanned aerial vehicle is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial equipment inspection, and in particular to a method and device for autonomous flight inspection of a UAV, an electronic device and a storage medium. BACKGROUND

[0002] With the development of science and technology, the inspection technology has also made great progress. From traditional manual inspection to UAV inspection, the efficiency and safety of inspection have been improved, and areas that are difficult to reach or pay attention to by manual inspection have also been covered.

[0003] The existing UAV inspection technology, when detecting faults of industrial equipment, usually shoots visible light images from a single angle through an ordinary camera, and cannot obtain detailed information about the industrial equipment from other angles. Moreover, the target environment in which the UAV works is complex, and may have insufficient lighting, shadows, reflections and other situations, which affect the image quality of the visible light images shot by the camera, and the working state of the industrial equipment is easily misjudged or missed, thereby affecting the efficiency of UAV inspection. SUMMARY

[0004] Therefore, the present application provides a method and device for autonomous flight inspection of a UAV, an electronic device and a storage medium, which can improve the efficiency and accuracy of UAV inspection.

[0005] In a first aspect, the present application provides a method for autonomous flight inspection of a UAV, comprising:

[0006] obtaining a plurality of original visible light images, a plurality of infrared thermal imaging images and a plurality of ultraviolet imaging images under a plurality of viewing angles in a target environment;

[0007] performing stitching processing on the plurality of original visible light images to obtain a panoramic visible light image;

[0008] constructing a quantumized energy field according to the infrared thermal imaging images and the ultraviolet imaging images;

[0009] associating and mapping the quantumized energy field with the panoramic visible light image to obtain a first fusion image;

[0010] performing gradient processing on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images, adjusting pixel value distribution of the first fusion image according to gradient information obtained by the gradient processing, and obtaining a second fusion image;

[0011] converting the second fusion image to a target color space, performing color offset compensation on the converted second fusion image in the target color space, and obtaining a multi-modal target image;

[0012] inputting the multi-modal target image into a pre-configured fault diagnosis model to generate a device state detection report;

[0013] determining an inspection path and an inspection parameter of the unmanned aerial vehicle according to the device state detection report.

[0014] In a possible implementation, the converting the second fused image to a target color space further includes:

[0015] determining a first color space to which the second fused image belongs;

[0016] converting the second fused image from the first color space to the target color space by a color space conversion algorithm.

[0017] In a possible implementation, the processing the plurality of original visible light images to obtain a panoramic visible light image includes:

[0018] performing light compensation and enhancing local contrast on the plurality of original visible light images to obtain a plurality of visible light enhanced images;

[0019] performing stitching processing on the plurality of visible light enhanced images to obtain the panoramic visible light image.

[0020] In a possible implementation, the performing light compensation on the plurality of original visible light images to obtain a plurality of visible light enhanced images includes:

[0021] constructing a three-dimensional light intensity model according to the infrared thermal imaging image and the ultraviolet imaging image;

[0022] performing light compensation processing on the plurality of original visible light images by the three-dimensional light intensity model to obtain a plurality of visible light reconstructed images;

[0023] performing contrast enhancement processing on the plurality of visible light reconstructed images to obtain the plurality of visible light enhanced images.

[0024] In a possible implementation, the performing light compensation processing on the plurality of original visible light images by the three-dimensional light intensity model to obtain a plurality of visible light reconstructed images includes:

[0025] performing decomposition processing on any original visible light image in the plurality of original visible light images to obtain an original low-frequency subband and at least one high-frequency subband;

[0026] performing light compensation on the original low-frequency subband based on the three-dimensional light intensity model to obtain a first low-frequency subband;

[0027] performing local contrast enhancement on the first low-frequency subband to obtain a second low-frequency subband;

[0028] reconstructing the at least one high-frequency sub-band and the second low-frequency sub-band to obtain the reconstructed visible light image.

[0029] In a possible implementation, the splicing the plurality of visible light enhanced images to obtain the panoramic visible light image comprises:

[0030] performing feature detection on a first visible light enhanced image and a second visible light enhanced image in the plurality of visible light enhanced images to obtain a first feature point set of the first visible light enhanced image and a second feature point set of the second visible light enhanced image, the first visible light enhanced image and the second visible light enhanced image being respectively based on two continuously acquired original visible light images;

[0031] constructing a geometric constraint model based on shooting parameters of the plurality of visible light enhanced images, and matching feature points in the first feature point set and the second feature point set through the geometric constraint model to obtain a plurality of first feature point pairs;

[0032] obtaining image change parameters based on the plurality of first feature point pairs;

[0033] performing affine transformation and fusion on the plurality of visible light enhanced images based on the image change parameters to obtain a panoramic visible light image.

[0034] In a possible implementation, the adjusting, according to the device state detection report, a patrol path and patrol parameters of a UAV patrol operation comprises:

[0035] obtaining a failure area and a risk area in the target environment according to the device state monitoring report, and determining a patrol path of the UAV according to the failure area and the risk area, the patrol path passing through the failure area and the failure area;

[0036] obtaining device failure information in the target environment according to the device state monitoring report, and determining patrol parameters of the UAV according to the failure device information, the patrol path comprising imaging device parameters and flight parameters.

[0037] In a second aspect, an embodiment of the present application provides a UAV autonomous flight patrol device, comprising:

[0038] an acquisition module configured to acquire a plurality of original visible light images, a plurality of infrared thermal imaging images, and a plurality of ultraviolet imaging images in a plurality of perspectives in a target environment;

[0039] a splicing module configured to splice the plurality of original visible light images to obtain a panoramic visible light image;

[0040] a construction module configured to construct a quantized energy field according to the infrared thermal imaging images and the ultraviolet imaging images;

[0041] a mapping module configured to associate and map the quantized energy field with the panoramic visible light image to obtain a first fusion image;

[0042] a fusion module configured to perform gradient processing on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images, and to adjust pixel value distribution of the first fusion image according to gradient information obtained by the gradient processing to obtain a second fusion image;

[0043] a color cast compensation module configured to convert the second fusion image to a target color space, and to perform color cast compensation on the converted second fusion image in the target color space to obtain a multi-modal target image;

[0044] a diagnosis module configured to input the multi-modal target image into a pre-configured fault diagnosis model to generate a device state detection report;

[0045] a determination module configured to determine an inspection path and an inspection parameter of the unmanned aerial vehicle according to the device state detection report.

[0046] In a third aspect, an electronic device is provided, and has the characteristics that it includes at least one processor, at least one memory, a communication interface, and a lead wire.

[0047] The memory stores a computer program, and the processor implements the steps of the unmanned aerial vehicle autonomous flight inspection method when executing the computer program stored in the memory.

[0048] The processor, the memory, and the communication interface are connected through the lead wire to realize data transmission.

[0049] In a fourth aspect, a computer readable storage medium is provided, and has the characteristics that it stores corresponding instructions, and the corresponding instructions can make a computer implement the steps of the unmanned aerial vehicle autonomous flight inspection method.

[0050] The unmanned aerial vehicle autonomous flight inspection method, device, electronic equipment and storage medium provided in the application obtain panoramic visible light images by splicing each original visible light image under different perspectives obtained in the unmanned aerial vehicle inspection process, construct a quantized energy field according to an infrared thermal imaging image and an ultraviolet imaging image, process the panoramic visible light image by using the quantized energy field to obtain a first fusion image; perform gradient processing on the infrared thermal imaging image and the ultraviolet imaging image, adjust pixel values of the first fusion image by using gradient information to obtain a second fusion image, and finally obtain a multi-modal target image after color space conversion and color cast compensation of the second fusion image. The image quality of the obtained multi-modal target image is greatly improved compared with the original visible light image, the details of industrial equipment in the target environment can be more clearly reflected, the multi-modal target image is analyzed by using a fault diagnosis model, the positioning accuracy of fault equipment and risk equipment in the target environment is greatly improved, the accuracy of industrial equipment fault degree judgment is significantly improved, and the efficiency of unmanned aerial vehicle inspection is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are included to provide a further understanding of the application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0052] Figure 1 Fig. 1 shows a flowchart of one of the unmanned aerial vehicle autonomous flight inspection methods according to an embodiment of the application;

[0053] Figure 2 Fig. 2 shows a flowchart of splicing processing of multiple original visible light images to obtain panoramic visible light images in the unmanned aerial vehicle autonomous flight inspection method according to an embodiment of the application;

[0054] Figure 3 Fig. 3 shows a flowchart of converting the second fusion image to a target color space in the unmanned aerial vehicle autonomous flight inspection method according to an embodiment of the application;

[0055] Figure 4 Fig. 4 shows a structural block diagram of the unmanned aerial vehicle autonomous flight inspection device according to an embodiment of the application;

[0056] Figure 5 Fig. 5 shows a structural block diagram of the electronic equipment according to an embodiment of the application. DETAILED DESCRIPTION

[0057] With reference to the drawings, a kind of unmanned aerial vehicle autonomous flight inspection method provided by specific embodiments and its application scene of the application are described in detail, obviously, the described embodiments are part of the embodiments of the application, not all embodiments, under the condition of no conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the present application.

[0058] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a kind, and the number of objects is not limited, for example, the first object can be one or more. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally indicates that the objects before and after are in a "or" relationship.

[0059] Inspection of industrial equipment in the target environment is the core to ensure the normal operation of industrial equipment, and the accuracy of its inspection results is directly related to stability and safety. With the development of science and technology, the inspection method has also changed. Unmanned aerial vehicle inspection technology has become an important tool for unmanned inspection due to its efficiency and safety, but the image processing technology of related technology still has obvious deficiencies when dealing with different environments.

[0060] Currently, the unmanned aerial vehicle inspection technology mainly obtains visible light images by single-angle shooting of industrial equipment in the target environment through ordinary cameras, which cannot fully capture the details of the industrial equipment in the target environment, and the light conditions in the target environment are complex (such as shadow, reflection, etc.), Therefore, it will affect the image quality of the visible light image shot by the camera, so as to miss or misjudge some more hidden industrial equipment failures, affecting the efficiency of inspection.

[0061] Based on the above, the present application provides an unmanned aerial vehicle autonomous flight inspection method, which improves the efficiency and accuracy of inspection and ensures the stable and efficient operation of industrial equipment in the target environment.

[0062] As shown in Figure 1 The execution subject of the method provided by the embodiment of the present application can be an electronic device in a server, a server group or a cloud server. In this embodiment, the execution subject is taken as an example of the control device of the transformer substation control system; of course, those skilled in the art can also run the method of the present disclosure according to the needs in other systems, and the embodiments of the present application do not make special limitation.

[0063] Step 101 : Acquire a plurality of original visible light images, a plurality of infrared thermal imaging images, and a plurality of ultraviolet imaging images at a plurality of viewing angles in a target environment.

[0064] The target environment can be an indoor environment of a substation, or an indoor or outdoor environment in other industrial scenarios where multiple industrial equipment are placed. The embodiments of the present application do not specifically limit this.

[0065] The original visible light images from multiple perspectives can be images of a certain area or a certain industrial equipment acquired by a high-resolution visible light camera carried by a drone at different heights and directions in the same area, or they can be images of a certain area or a certain industrial equipment acquired at different heights and directions in different areas within the target environment.

[0066] The drone can be equipped with a high-resolution visible light camera, an infrared imager, and an ultraviolet imager. The drone can fly along a preset initial inspection path in the target environment, and obtain multiple original visible light images, multiple infrared thermal imaging images, and multiple ultraviolet imaging images according to pre-set time intervals and distance intervals.

[0067] The original visible light image can be used to reflect the appearance characteristics of industrial equipment in the target environment; the infrared thermal imaging image is an image obtained by the infrared imager carried by the drone, which is used to reflect the thermal radiation data in the target environment; the ultraviolet imaging image is an image obtained by the ultraviolet imager carried by the drone, which is used to reflect the high-frequency electromagnetic field information in the target environment.

[0068] Step 102 : stitching the multiple acquired original visible light images to obtain a panoramic visible light image.

[0069] After acquiring multiple raw visible light images, the control device can perform feature point matching on these images from multiple perspectives, as lighting conditions vary across different locations in the target environment (such as potential shadows and reflections). Based on the matching results, the images are stitched together to create a panoramic visible light image of the target environment. This panoramic visible light image contains comprehensive image information about the industrial equipment in the target environment.

[0070] Step 103: construct a quantized energy field based on the infrared thermal imaging image and the ultraviolet imaging image.

[0071] For example, from the infrared thermal imaging image I IR Get the temperature value T corresponding to a pixel point (x, y) IR (x, y), from the UV imaging image I UV The discharge photon count N at this position is countedUV (x, y). The quantumized energy field about the target environment is constructed by using a first formula, which is:

[0072] E(x, y) = N UV (x, y) h v + t T IR (x, y) 4

[0073] wherein E(x, y) represents the value of the quantumized energy field at (x, y); (x, y) represents the horizontal and vertical coordinates of the pixel point in the infrared thermal imaging image plane, so as to determine the specific position of the pixel point at (x, y) in the infrared thermal imaging image and the ultraviolet imaging image; N UV represents the number of ultraviolet discharge photons in a unit area of coordinates; h represents the Planck constant; v represents the ultraviolet photon frequency; t represents the Stefan-Boltzmann constant; T IR represents the temperature value corresponding to the infrared thermal radiation at the coordinate (x, y).

[0074] Exemplarily, in the target environment, at the pixel point (10, 10), the discharge photon count N UV (10, 10) = 100 of the ultraviolet imaging image is counted, the temperature value T IR (10, 10) = 300 K of the pixel point is obtained from the infrared thermal imaging image, the ultraviolet photon frequency v = 1015 Hz, and E(10, 10) = 100 x 6.63 x 10-34 x 1015 + 5.67 x 10-8 x 3004 = 6.63 x 10-17 + 4.59 is obtained by substituting the above formula. By performing the above calculation on each pixel point in the infrared thermal imaging image and the ultraviolet imaging image, the entire quantumized energy field distribution is obtained, and then the quantumized energy field is constructed.

[0075] In step 104, the quantumized energy field is associated and mapped with the panoramic visible light image to obtain a first fusion image.

[0076] Specifically, the quantumized energy field E(x, y) is mapped to the visible light pixel space by using a second formula, the pixel value of each pixel point in the panoramic visible light image is calculated, the pixel value of each point obtained by calculation is substituted for the pixel value of the corresponding pixel point in the panoramic visible light image, and thus the first fusion image is obtained. The second formula is:

[0077]

[0078] wherein I fusion1 (x, y) represents the pixel value of the first fusion image at (x, y); represents I visible(x, y) is the pixel value of the panoramic visible light at (x, y); k is an adjustable coefficient for controlling the sensitivity of the mapping; E(x, y) represents the value of the quantized energy field at (x, y); E0 is the reference value of the energy field.

[0079] Exemplarily, for a pixel point (20, 20) in the panoramic visible light image, its pixel value I visible (20, 20) = 150, the corresponding quantized energy field value E(20, 20) of the point is 5, assuming k = 1 and E0 = 3, then The above calculation is performed on each pixel point of the panoramic visible light image to obtain a first fusion image I fusion1 .

[0080] In step 105, gradient processing is performed on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images to obtain target gradient information, and the pixel values of the pixel points in the first fusion image are adjusted according to the target gradient information to obtain a second fusion image.

[0081] Specifically, gradient calculation is performed on the infrared thermal imaging image I IR and the ultraviolet imaging image I UV respectively to obtain corresponding first gradient information of the infrared thermal imaging image I IR and second gradient information of the ultraviolet imaging image I UV . The two gradient information are merged to obtain target gradient information.

[0082] Exemplarily, the control device can use a gradient operator (such as a Sobl operator) to calculate gradient amplitudes and , that is, to obtain the first gradient information and the second gradient information. Then, a Poisson equation is solved in the gradient field, and by solving the equation, target gradient information is obtained. The target gradient information is fused into the related information of the first fusion image to obtain a second fusion image. The Poisson equation is:

[0083]

[0084] wherein, represents the gradient amplitude of the infrared thermal imaging image, represents the gradient amplitude of the ultraviolet imaging image, and Φ and ω are weight coefficients which can be adjusted according to actual conditions.

[0085] Exemplarily, the gradient amplitude of a certain region in the infrared thermal imaging image is large, indicating that the temperature change of the region is obvious. By solving the above Poisson equation, the obtained target gradient information of the temperature change is fused into the related information of the first fusion image to adjust the pixel values of the pixel points in the first fusion image, so that the second fusion image can more obviously reflect the temperature change characteristics of the region.

[0086] Step 106, converting the second fusion image to a target color space, performing color cast compensation on the converted second fusion image in the target color space to obtain a multi-modal target image.

[0087] The second fusion image is converted to a target color space, and the chrominance and / or brightness of the second fusion image is compensated in the target color space. Specifically, the chrominance component and / or brightness component of each pixel point in the second fusion image can be adjusted according to a third formula, so as to realize chrominance compensation and brightness compensation of each pixel point in the second fusion image. After the above adjustment of each pixel point in the second fusion image, a multi-modal target image after color cast compensation of the second fusion image is obtained. The multi-modal target image is an image that reflects more comprehensive information of industrial equipment in the target environment, which is obtained by fusing the information presented in the multiple infrared thermal imaging images and the multiple ultraviolet imaging images with the information presented in the panoramic visible light image. The above third formula is:

[0088]

[0089] wherein, Δa ye represents the updated chrominance component value of the pixel point calculated, represents the chrominance component value a ye of the panoramic visible light in the target color space, represents the chrominance component value a ye of the second fusion image in the target color space, and δ is dynamically determined according to the original visible light image histogram.

[0090] Exemplarily, by calculating the statistical characteristics of the original visible light histogram, δ=0.8 is determined, and at a certain pixel point, Δaye=10-0.8x8=3.6, and the a ye component value of the pixel point is updated to 3.6. The above calculation is performed on all pixel points of the second fusion image, and finally a multi-modal target image after color cast compensation is obtained.

[0091] Step 107, inputting the multi-modal target image into a pre-configured fault diagnosis model to generate a device state detection report.

[0092] Specifically, using a deep learning framework (such as TensorFlow or PyTorch), a suitable neural network structure (such as a convolutional neural network CNN) is selected, and a plurality of pre-collected sample images in the target environment are used to train the neural network structure. The sample images include multi-modal images of the industrial equipment in normal state and multi-modal images of the industrial equipment in various fault states, which are input into the selected neural network structure for training to obtain a first fault diagnosis model. Through a back propagation algorithm, the model parameters are continuously adjusted to minimize the error between the predicted fault type and degree of the industrial equipment in the target environment by the first fault diagnosis model and the actual label, so as to obtain a second fault diagnosis model, i.e., a pre-configured fault diagnosis model.

[0093] The multi-modal target image is input into the pre-configured fault diagnosis model, and the pre-configured fault diagnosis model outputs a device state detection report according to a preset device state detection report template.

[0094] In step 108, the inspection path and inspection parameters of the unmanned aerial vehicle are determined according to the device state detection report.

[0095] Specifically, the device state detection report can include the name, location, fault type, fault degree, and possible impact and suggestions of the industrial equipment, and specific measures, etc. The inspection path and inspection parameters of the unmanned aerial vehicle are adjusted according to the information in the device detection report. For example, according to the location of the fault equipment in the device fault detection report, the flight height of the unmanned aerial vehicle during inspection is adjusted to obtain clearer images of the fault equipment, and the inspection frequency of the area where the fault equipment is located is increased to obtain more timely information of the fault equipment.

[0096] The method in the embodiment obtains panoramic visible light images by splicing each original visible light image obtained in different perspectives, constructs a quantumized energy field according to the infrared thermal imaging image and the ultraviolet imaging image, associates and maps the quantumized energy field and the panoramic visible light image to obtain a first fusion image, performs gradient processing on the infrared thermal imaging image and the ultraviolet imaging image, adjusts the pixel value of the first fusion image using the gradient information obtained through the gradient processing to obtain a second fusion image, and finally obtains a multi-modal target image by performing color space conversion and color cast compensation on the second fusion image. The image quality of the obtained multi-modal target image is greatly improved compared with the original visible light image, and the details of the industrial equipment in the target environment can be more clearly presented. Using the fault diagnosis model to analyze the multi-modal target image greatly improves the accuracy of positioning the fault equipment and the risk equipment in the target environment, significantly improves the precision of judging the fault degree of the industrial equipment, and greatly improves the efficiency of the unmanned aerial vehicle inspection.

[0097] In some embodiments, as Figure 2As shown, step 102 can be specifically implemented by steps 1021 to 1022.

[0098] Step 1021, light compensation and local contrast enhancement are performed on the acquired multiple original visible light images to obtain multiple visible light enhanced images.

[0099] Specifically, the brightness of the original visible light image is improved through light compensation, and the blurred details in the original visible light image become clearer through local contrast enhancement, solving the problem of blurred original visible light images caused by the complexity of the target environment.

[0100] Step 1022, the multiple visible light enhanced images are stitched to obtain a panoramic visible light image.

[0101] Exemplarily, the multiple visible light enhanced images are matched with feature points, and the visible light enhanced images are stitched according to the matched feature points to obtain a panoramic visible light image, so as to realize comprehensive acquisition of the surface information of the industrial equipment.

[0102] In specific embodiments, step 1021 is implemented by steps 10211 to 10213.

[0103] Step 10211, a three-dimensional light intensity model is constructed according to the infrared thermal imaging image and the ultraviolet imaging image.

[0104] Specifically, the control device acquires the thermal radiation data IR(x, y) at (x, y) in the target environment through the infrared sensor carried on the unmanned aerial vehicle, and acquires the high-frequency electromagnetic field information UV(x, y) at (x, y) in the target environment through the ultraviolet imaging device carried on the unmanned aerial vehicle. The light intensity at (x, y) is calculated according to the fourth formula for constructing a three-dimensional light intensity model. The fourth formula is:

[0105] L(x, y) = a IR(x, y) + b UV(x, y) e -γ·D(x，y)

[0106] Where (x, y) represents the spatial coordinates of the unmanned aerial vehicle in the target environment, L(x, y) represents the light intensity, IR(x, y) represents the thermal radiation data at (x, y) in the target environment acquired by the infrared sensor; UV(x, y) represents the high-frequency electromagnetic field information at (x, y) in the target environment acquired by the ultraviolet imaging device; D(x, y) represents the distance attenuation factor, which is used to reflect the attenuation of light intensity with the increase of distance; a and b represent the weight coefficients, and g represents the attenuation coefficient, a, b, and g can be obtained through experimental data or machine learning algorithm.

[0107] Exemplarily, in the target environment, the unmanned aerial vehicle collects thermal radiation data and high-frequency electromagnetic field information at different positions. For a position (x0, y0), the thermal radiation data measured by the infrared sensor is IR(x0, y0) = 100, the high-frequency electromagnetic field information measured by the ultraviolet imaging device is UV(x0, y0) = 50, the distance attenuation factor is D(x0, y0) = 0.5, and the weight coefficients α = 0.6, β = 0.4, and γ = 0.2 are finally obtained through experimental data or machine learning algorithms. Then, the illumination intensity of the position is calculated according to the three-dimensional illumination intensity model as follows:

[0108] In step 10212, the original visible light images are light compensated by the three-dimensional illumination intensity model to obtain a plurality of visible light reconstructed images.

[0109] Specifically, the illumination intensity of the original visible light image is calculated according to the three-dimensional illumination intensity model, the original visible light image is decomposed, the illumination intensity of the decomposed original low-frequency subband is adjusted, and then the new image, i.e., the visible light reconstructed image, is obtained by recombining. Compared with the original visible light image, the clarity of the visible light reconstructed image is significantly improved, solving the problem of blurred original visible light images caused by factors such as shadows and reflections in the target environment.

[0110] In specific embodiments, step 10212 is implemented by steps 10212a to 10212d.

[0111] In step 10212a, any original visible light image in the plurality of original visible light images is decomposed to obtain an original low-frequency subband and at least one high-frequency subband.

[0112] Specifically, the original visible light image I(x, y) is decomposed into a low-frequency subband L original and three high-frequency subbands, i.e., horizontal high-frequency H x , vertical high-frequency H y , and diagonal high-frequency H xy , by using discrete wavelet transform (DWT) and a set of high-pass filters and low-pass filters. The low-frequency subband mainly contains the illumination component of the original visible light image, reflecting the overall brightness and background information of the original visible light image; the high-frequency subband mainly contains the texture details of the original visible light image, wherein the horizontal high-frequency H x reflects the horizontal object edge, image texture, and other information, the vertical high-frequency H y mainly reflects the vertical object edge, image texture, and other information, and the diagonal high-frequency H xy mainly reflects the diagonal object edge, image texture, and other information.

[0113] Step 10212b, illuminance compensation is performed on the original low-frequency sub-band according to the three-dimensional illuminance intensity model to obtain a first low-frequency sub-band.

[0114] Specifically, the original low-frequency sub-band is compensated for illuminance according to the constructed three-dimensional illuminance intensity model and a preset ideal illuminance intensity. The illuminance intensity of the first low-frequency sub-band is calculated by a fifth formula. The fifth formula is:

[0115]

[0116] wherein, L comp represents the illuminance intensity of the first low-frequency sub-band, L original represents the illuminance intensity of the original low-frequency sub-band, L target represents the preset ideal illuminance intensity, L model represents the illuminance intensity value calculated at the corresponding position according to the illuminance intensity model.

[0117] Exemplarily, the illuminance intensity value L original = 50 of the original low-frequency sub-band of a certain original visible light image in the target environment is calculated by the three-dimensional illuminance intensity model to obtain the illuminance intensity L model = 40 at the position of acquiring the original visible light image in the target environment, and the preset ideal illuminance value L target = 60, then the illuminance intensity of a certain pixel point in the first low-frequency sub-band is By performing the above calculation on each pixel point in the original low-frequency sub-band, the illuminance compensation of the entire original low-frequency sub-band is realized to obtain the first low-frequency sub-band.

[0118] Step 10212c, local contrast enhancement is performed on the first low-frequency sub-band to obtain a second low-frequency sub-band.

[0119] Specifically, the first low-frequency sub-band is subjected to local contrast enhancement by using a segmented Sigmoid function, and the pixel value of each pixel point in the second low-frequency sub-band is calculated according to a sixth formula. The sixth formula is:

[0120]

[0121] wherein, V out represents the pixel value of any pixel point in the second low-frequency sub-band, V in represents the pixel value of the pixel point in the corresponding first low-frequency sub-band, e represents a natural constant, and the parameters k and c are dynamically determined by the entropy value of the image histogram of the first low-frequency sub-band.

[0122] Further, the entropy value H of the first low-frequency sub-band image histogram is calculated, and the values of the parameters k and c are determined according to a preset mapping relationship. If the entropy value H is small, it indicates that the image gray scale distribution is relatively concentrated, and at this time, the value of the parameter k is increased to enhance the contrast stretching effect; if the entropy value H is large, the value of the parameter k is appropriately reduced.

[0123] Exemplarily, the pixel value V of a certain pixel point in the first low-frequency sub-band in = 0.3, the parameter k = 2 and the parameter c = 0.2 are determined by the entropy value of the image histogram of the first low-frequency sub-band, and then the pixel value V of the corresponding pixel point in the second low-frequency sub-band The above calculation is performed on all pixel points of the first low-frequency sub-band to obtain the second low-frequency sub-band.

[0124] Step 10212d, reconstructing at least one high-frequency sub-band and the second low-frequency sub-band to obtain a visible light reconstructed image.

[0125] Specifically, the second low-frequency sub-band is inversely wavelet transformed (IDWT) with three high-frequency sub-bands (horizontal direction high-frequency H x , vertical direction high-frequency H y , and diagonal direction high-frequency H xy ), that is, the information of the second low-frequency sub-band and the three high-frequency sub-bands is recombined to realize image reconstruction through the operation of the inverse filter corresponding to the wavelet transform, to obtain the visible light reconstructed image. After the above steps, the clarity of the original visible light image is significantly improved.

[0126] Step 10213, performing contrast enhancement processing on a plurality of visible light reconstructed images to obtain a plurality of visible light enhanced images.

[0127] Specifically, an improved Laplacian operator is used to perform edge enhancement on the visible light reconstructed image. The seventh formula used for calculation is:

[0128]

[0129] Wherein, G(x, y) represents the edge enhancement value of the pixel point at coordinate (x, y) in the visible light reconstructed image, I(x, y) represents the pixel point at coordinate (x, y) in the visible light reconstructed image, represents the gradient of the pixel point at (x, y) in the visible light reconstructed image, represents the Laplacian value of a pixel point at (x, y) in the visible light reconstructed image, and σ represents a gradient adaptive coefficient. The value of σ can be dynamically adjusted according to the statistical information of the local gradient of the image. In the area where the gradient changes greatly, the value of σ can be appropriately increased to enhance the effect of edge enhancement processing. The above calculation is performed on each pixel point of the visible light reconstructed image to obtain a visible light enhanced image, which is convenient for subsequent stitching of the visible light enhanced image to obtain a panoramic visible light image.

[0130] In some embodiments, step 1022 is implemented by steps 10221 to 10224:

[0131] Step 10221, feature detection is performed on a first visible light enhanced image and a second visible light enhanced image in the plurality of visible light enhanced images to obtain a first feature point set of the first visible light enhanced image and a second feature point set of the second visible light enhanced image.

[0132] The first visible light enhanced image and the second visible light enhanced image are respectively obtained based on two continuously acquired original visible light images. For the two continuously acquired original visible light images, there are more similar feature points in the first feature point set and the second feature point set, which is convenient for subsequent feature point matching and image stitching.

[0133] In this way, the panoramic visible light image of the target environment is finally obtained by stitching the plurality of original visible light images acquired in the plurality of perspectives in the unmanned aerial vehicle inspection process.

[0134] Specifically, each visible light enhanced image is divided into a plurality of sub-region images and numbered, and then the improved Harris corner detection algorithm is used for feature detection on each sub-region image.

[0135] The improved Harris corner detection algorithm introduces the regional gray variance as a weight factor, and the feature point response function is:

[0136]

[0137] wherein R new represents the response function obtained by calculation; R old represents the response function of the traditional Harris algorithm; ε 2 represents the regional gray variance, reflecting the dispersion degree of the gray scale in the region; μ represents the average gray scale of the region.

[0138] The response function value of each pixel point in the visible light enhanced image is calculated, a suitable threshold is set according to the calculation result, and the points with the response function value greater than the threshold obtained by calculation are screened out as feature points, and the coordinates and the number of the sub-region where the feature points are located are recorded.

[0139] Exemplarily, in a certain sub-region, if the response function values of 5 pixel points are greater than the set threshold value after calculation, the 5 pixel points are determined as the feature points of the sub-region.

[0140] At step 10222, a geometric constraint model is constructed based on the shooting parameters of the plurality of visible light enhanced images; and the feature points in the first feature point set and the second feature point set are matched through the geometric constraint model to obtain a plurality of first feature point pairs.

[0141] In the formula, one similar feature point in the first feature point set and the second feature point set is one first feature point pair, and a plurality of identical feature points in the first feature point set and the second feature point set are a plurality of first feature point pairs. The plurality of first feature point pairs obtained by matching the feature points in the first feature point set and the second feature point set will be used to obtain corresponding parameters subsequently.

[0142] Specifically, according to the shooting position and the shooting angle when the original visible light image corresponding to the visible light enhanced image is acquired, such as collecting the three-dimensional coordinates, the attitude angle (including the pitch angle, the yaw angle, and the roll angle) and other information of the unmanned aerial vehicle when the original visible light image is acquired through the positioning device and the attitude sensor carried by the unmanned aerial vehicle, the geometric constraint model is established according to the collected information, and the feature point matching is performed by using the epipolar constraint principle in the geometric constraint model.

[0143] For the feature point p a in the visible light enhanced image A, according to the epipolar geometry relationship in the geometric constraint model, the epipolar line l a of the feature point p j in the visible light enhanced image B is calculated. The epipolar line can be calculated according to the eighth formula. Through the epipolar geometry relationship, the essential matrix E or the fundamental matrix F (which can be acquired by camera calibration) of the visible light enhanced image is used to realize this process, which will not be described herein. Then, the matching range of the feature point p a in the visible light enhanced image B is limited to the vicinity of the epipolar line l j , and a plurality of first feature point pairs are obtained. The eighth formula is as follows:

[0144] l j =F·p a

[0145] In the formula, p a represents the feature point in the visible light enhanced image A, l j represents the epipolar line of the feature point p a in the visible light enhanced image B, and F represents the fundamental matrix of the visible light enhanced image.

[0146] Exemplarily, in the epipolar line l jSearch for the feature point p within a certain distance (such as 5 pixels) on both sides a If the matching feature point is found within the range a Points p with similar features b , then (p a , p b ) are recorded as the first feature point pair. Repeat the above operation for all feature points in the visible light enhanced image A to obtain multiple sets of first feature point pairs.

[0147] Step 10223: Obtain image change parameters of the visible light enhanced image based on the first feature point pair.

[0148] Specifically, a feature descriptor is constructed for each set of feature point pairs, and each feature descriptor is combined with the gradient direction histogram and grayscale moment features of a certain area around each set of feature point pairs. For each feature point in each set of feature point pairs, a gradient direction histogram is calculated in a certain area around it (such as a 15×15 pixel window centered on the feature point), the gradient direction is divided into several gradient intervals, and the sum of the gradient amplitudes in each gradient interval is counted to obtain the gradient direction histogram feature vector. The grayscale moment features of a certain area around each feature point in each set of feature point pairs are calculated, such as the zero-order moment m 00 =Σ x，y I(x，y) (used to calculate the center of mass of the area), first-order moment m 10 =Σ x，y xI(x，y) 、m 01 =Σ x，y yI(x，y) Etc., the calculated multiple grayscale moment features are combined into a new feature descriptor. For example, for the feature point pair (p a , p b ), whose feature descriptors are d a and d b , calculate the Euclidean distance between the feature descriptors of each first feature point pair. The calculation formula of Euclidean distance is:

[0149]

[0150] Where D represents the Euclidean distance, n represents the total number of dimensions of the feature descriptor, i represents the dimension index of the current calculation, and d ai Represents a feature point pair (p a , p b ) a The i-th feature descriptor, d bi Represents a feature point pair (p a , p b ) b The i-th feature descriptor of .

[0151] A threshold is set according to actual conditions, and the first feature point pairs with the Euclidean distance less than the threshold are selected as the second feature point pairs.

[0152] Exemplarily, in 100 groups of first feature point pairs, the Euclidean distances of 30 groups of first feature point pairs are less than the threshold after calculation and screening, and then the 30 groups of first feature point pairs are the second feature point pairs.

[0153] According to the affine transformation model where a1, b1, c1, d1 control the rotation, scaling and shearing of the image, t x , t y control the translation of the image. The obtained second feature point pairs are used to fit the affine transformation model T by using the least square method. Assuming that there are n groups of second feature point pairs, for the i-th group of second feature point pairs (p 1i , p 2i ), where p 1i =(x 1i , y 1i , 1) T , p 2i =(x 2i , y 2i , 1) T , the transformation parameters a1, b1, c1, d1, t x , t y are determined by minimizing the error function . In the specific calculation, the error function is expanded and the partial derivatives of each parameter are obtained, and the partial derivatives are set to 0 to obtain a linear equation group, and the values of the transformation parameters are obtained by solving the linear equation group.

[0154] Exemplarily, if there are 5 groups of second feature point pairs, the image transformation parameters a1=1.2, b1=-0.1, c1=0.2, d1=1.1, t x =5, t y =3 are obtained after calculation and solving of the linear equation group.

[0155] In step 10224, affine transformation and fusion are performed on the plurality of visible light enhanced images based on the image change parameters to obtain a panoramic visible light image.

[0156] Specifically, affine transformation is performed on the plurality of visible light enhanced images according to the image transformation parameters. Specifically, affine transformation matrix T is used for each pixel point in each visible light enhanced image, such as p' = T p, (wherein p' is the pixel point coordinate after transformation, and p is the original pixel point coordinate). The visible light enhanced images after affine transformation are fused. In the overlapping area, the pixel values of the pixel points in the visible light enhanced images are fused by the ninth formula with the distance to the boundary of the visible light enhanced image as the weight. The above operation is repeated for the pixel points in the overlapping area, and finally the panoramic visible light image is obtained. The ninth formula is:

[0157]

[0158] wherein (x, y) represents the spatial coordinates of a pixel point in the overlapping area of the visible light enhanced images I1 and I2, I final (x, y) represents the pixel value of the panoramic visible light image at the (x, y) position, I1(x, y) and I2(x, y) represent the pixel values of the two adjacent acquired original visible light images at the (x, y) position, and d1 and d2 respectively represent the distances of the pixel point to the boundaries of the visible light enhanced images I1 and I2.

[0159] Exemplarily, for a pixel point (x0, y0) in a certain overlapping area, d1 = 10, d2 = 5, I1(x0, y0) = 100, and I2(x0, y0) = 120, then the fused pixel value is

[0160] The panoramic visible light image is spliced based on the visible light enhanced images, and compared with the original visible light image, the clarity of the panoramic visible light image is greatly improved, and the texture and contour of the industrial equipment in the target environment embodied in the panoramic visible light image are significantly improved.

[0161] In specific embodiments, as shown in FIG. 6, the conversion of the second fused image to the target color space in step 106 can be implemented by the following steps 1061 to 1063: Figure 3

[0162] Step 1061, determining the first color space to which the second fused image belongs.

[0163] Exemplarily, common color spaces include RGB color space, CMYK color space, sRGB color space, etc.

[0164] Step 1062, converting the second fused image from the first color space to the target color space by a color space conversion algorithm.

[0165] ​Specifically, the second fusion image is converted from the first color space to a target color space by using a standard color space conversion algorithm. The first color space can be an RGB color space, and the second color space can be a CIELab color space.

[0166] The conversion of the second fusion image into the color space can realize independent adjustment of the brightness and color of the second fusion image, and avoid mutual interference. In addition, the target color space is more suitable for processing images in a shadow or mixed light source scene, and color cast compensation of the second fusion image in the target color space can greatly improve the image quality of the second fusion image.

[0167] In specific embodiments, the step 108 can be implemented by the following steps 1081 to 1083.

[0168] In step 1081, the fault area and the risk area in the target environment are obtained according to the device state monitoring report.

[0169] Specifically, the target environment is divided into multiple areas and numbered, the fault area refers to a specific area where the industrial equipment with faults is located, and the risk area refers to a specific area where the industrial equipment with fault risks is located.

[0170] In step 1082, the inspection path of the unmanned aerial vehicle is determined according to the fault area and the risk area, and the inspection path passes through the fault area and the risk area.

[0171] For example, the sub-paths for inspecting the industrial equipment in the fault area and the risk area are determined according to the obtained fault area and risk area, the multiple sub-paths are combined, and the path for inspecting the industrial equipment in the target environment is determined.

[0172] In step 1083, the fault equipment information in the target environment is obtained according to the device state monitoring report, and the inspection parameters of the unmanned aerial vehicle are determined according to the fault equipment information. The inspection parameters include imaging equipment parameters and flight parameters.

[0173] Specifically, the device state detection report is analyzed, if the device state detection report shows that there is a fault or potential risk in the industrial equipment in a certain area in the target environment, the inspection frequency of the area is increased, the time interval between two inspections is shortened and the inspection path is adjusted; if the device state detection report shows that the industrial equipment is in good condition, the inspection frequency of the area can be appropriately reduced, and the time interval between two inspections is expanded. The inspection parameters of the unmanned aerial vehicle are adjusted according to the equipment fault information obtained by the inspection, including imaging equipment parameters (such as temperature resolution of an infrared thermal imager, photon detection sensitivity of an ultraviolet imaging device, etc.) and flight parameters (such as hovering time, flight speed, etc.), so that the unmanned aerial vehicle approaches the fault equipment when passing through the area next time, and hovers at each preset position to obtain the image of the fault equipment.

[0174] For example, if the temperature of the industrial equipment is too high, the control device can increase the temperature resolution of the infrared thermal imager carried on the unmanned aerial vehicle to more accurately detect the temperature change of the industrial equipment.

[0175] For example, the unmanned aerial vehicle performs inspection according to the adjusted new path, and hovers at different heights above the fault equipment and the risk equipment for 2 seconds respectively for the imaging collection device to collect images.

[0176] In this embodiment, the control device can automatically adjust the inspection path and the inspection parameters of the unmanned aerial vehicle in the target environment by analyzing the device state detection report, accurately position the fault equipment and the risk equipment, improve the frequency of collecting images of the fault equipment and the risk equipment, and change the angle of collecting images of the fault equipment and the risk equipment, so as to more accurately distinguish different fault types and their severity, while avoiding the problems of missed judgment or misjudgment, and improving the efficiency of the inspection process.

[0177] The unmanned aerial vehicle autonomous flight inspection method in the embodiments of the present application can be applied to an unmanned aerial vehicle autonomous flight inspection device. As shown in Figure 4 The unmanned aerial vehicle autonomous flight inspection device 400 comprises:

[0178] The acquisition module 401 is configured to acquire a plurality of original visible light images, a plurality of infrared thermal imaging images and a plurality of ultraviolet imaging images under a plurality of perspectives in a target environment;

[0179] The splicing module 402 is configured to splice the plurality of original visible light images to obtain a panoramic visible light image;

[0180] The construction module 403 is configured to construct a quantized energy field according to the infrared thermal imaging images and the ultraviolet imaging images;

[0181] The mapping module 404 is configured to associate and map the quantized energy field and the panoramic visible light image to obtain a first fusion image.

[0182] The fusion module 405 is configured to perform gradient processing on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images, adjust pixel value distribution of the first fusion image according to gradient information obtained through the gradient processing, and obtain a second fusion image.

[0183] The color cast compensation module 406 is configured to convert the second fusion image to a target color space, perform color cast compensation on the converted second fusion image in the target color space, and obtain a multi-modal target image.

[0184] The diagnosis module 407 is configured to input the multi-modal target image into a preconfigured fault diagnosis model, and generate a device state detection report.

[0185] The determination module 408 is configured to determine an inspection path and an inspection parameter of the unmanned aerial vehicle according to the device state detection report.

[0186] In some embodiments, the color cast compensation module 406 is further configured to determine a first color space to which the second fusion image belongs, and convert the second fusion image from the first color space to the target color space through a color space conversion algorithm.

[0187] In some embodiments, the stitching module 402 is further configured to perform light compensation and enhance local contrast on the plurality of original visible light images to obtain a plurality of visible light enhanced images, and perform stitching processing on the plurality of visible light enhanced images to obtain the panoramic visible light image.

[0188] In some embodiments, the stitching module 402 is further configured to obtain thermal radiation data of the target environment and high-frequency electromagnetic field information of the target environment, construct a three-dimensional light intensity model based on the thermal radiation data and the high-frequency electromagnetic field information, perform light compensation processing on the plurality of original visible light images through the three-dimensional light intensity model to obtain a plurality of visible light reconstructed images, and perform contrast enhancement processing on the plurality of visible light reconstructed images to obtain the plurality of visible light enhanced images.

[0189] In some embodiments, the stitching module 402 is further configured to perform decomposition processing on any original visible light image in the plurality of original visible light images to obtain an original low-frequency subband and at least one high-frequency subband, perform light compensation on the original low-frequency subband based on the three-dimensional light intensity model to obtain a first low-frequency subband, perform local contrast enhancement on the first low-frequency subband to obtain a second low-frequency subband, and reconstruct the at least one high-frequency subband and the second low-frequency subband to obtain the visible light reconstructed image.

[0190] In some embodiments, the splicing module 402 is further configured to perform feature detection on a first visible light enhanced image and a second visible light enhanced image in the plurality of visible light enhanced images to obtain a first set of feature points of the first visible light enhanced image and a second set of feature points of the second visible light enhanced image, the first visible light enhanced image and the second visible light enhanced image being respectively based on two continuously acquired original visible light images; construct a geometric constraint model based on shooting parameters of the plurality of visible light enhanced images; match feature points in the first set of feature points and the second set of feature points through the geometric constraint model to obtain a plurality of first feature point pairs; obtain image change parameters based on the plurality of first feature point pairs; and perform affine transformation and fusion on the plurality of visible light enhanced images based on the image change parameters to obtain a panoramic visible light image.

[0191] The embodiments of the present application further provide an electronic device, as shown in the figure, the electronic device 500 comprises a processor 501 and a memory 502, the memory 502 has a program or instruction which can be run on the processor 501, the program or instruction is executed by the processor 501 to realize each step of the embodiments of the above unmanned aerial vehicle autonomous inspection method, and the same technical effect can be achieved, to avoid repetition, here will not be repeated. Figure 5 The embodiments of the present application further provide an electronic device, as shown in the figure, the electronic device 500 comprises a processor 501 and a memory 502, the memory 502 has a program or instruction which can be run on the processor 501, the program or instruction is executed by the processor 501 to realize each step of the embodiments of the above unmanned aerial vehicle autonomous inspection method, and the same technical effect can be achieved, to avoid repetition, here will not be repeated.

[0192] The memory 502 can be used to store software programs and various data. The memory 502 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 502 can include a volatile memory or a non-volatile memory, or the memory 502 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 402 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0193] The processor 501 can include one or more processing units; optionally, the processor 501 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 501.

[0194] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize each process of the above-mentioned unmanned aerial vehicle autonomous inspection method embodiment, and the same technical effects can be achieved, to avoid repetition, which will not be described here.

[0195] The embodiment of the present application further provides a chip, which comprises a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running programs or instructions to realize the processes of the unmanned aerial vehicle autonomous inspection method and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0196] It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0197] The embodiment of the present application further provides a computer program product, which is stored in a storage medium, and is executed by at least one processor to realize the processes of the unmanned aerial vehicle autonomous flight inspection method and achieve the same technical effects. To avoid repetition, details are not repeated here.

[0198] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the present application is not limited to the order of functions shown or discussed, but can also include functions performed in a substantially simultaneous manner or in the opposite order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0199] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A method for autonomous flight inspection by a drone, characterized in that: include: Acquire multiple original visible light images, multiple infrared thermal imaging images, and multiple ultraviolet imaging images at multiple viewing angles of a target environment; performing stitching processing on the multiple original visible light images to obtain a panoramic visible light image; Constructing a quantized energy field according to the infrared thermal imaging image and the ultraviolet imaging image; Performing correlation mapping on the quantized energy field and the panoramic visible light image to obtain a first fused image; performing gradient processing on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images, and adjusting the pixel value distribution of the first fused image according to gradient information obtained by the gradient processing to obtain a second fused image; converting the second fused image into a target color space, and performing color deviation compensation on the converted second fused image in the target color space to obtain a multimodal target image; Inputting the multimodal target image into a preconfigured fault diagnosis model to generate a device status detection report; Determine the inspection path and inspection parameters of the drone based on the equipment status detection report.

2. The autonomous flight inspection method of a UAV according to claim 1, characterized in that: The converting the second fused image into a target color space further includes: determining a first color space to which the second fused image belongs; The second fused image is converted from the first color space to the target color space by using a color space conversion algorithm.

3. The autonomous flight inspection method of a UAV according to claim 1, characterized in that: The processing of the plurality of original visible light images to obtain a panoramic visible light image includes: performing light compensation and local contrast enhancement on the plurality of original visible light images to obtain a plurality of visible light enhanced images; The multiple visible light enhanced images are stitched together to obtain the panoramic visible light image.

4. The autonomous flight inspection method of a UAV according to claim 3, characterized in that: The performing light compensation on the plurality of original visible light images to obtain a plurality of visible light enhanced images includes: constructing a three-dimensional light intensity model based on the infrared thermal imaging image and the ultraviolet imaging image; Performing light compensation processing on the multiple original visible light images using the three-dimensional light intensity model to obtain multiple visible light reconstructed images; The multiple visible light reconstructed images are subjected to contrast enhancement processing to obtain the multiple visible light enhanced images.

5. The autonomous flight inspection method of a UAV according to claim 4, characterized in that: The light compensation processing of the plurality of original visible light images using the three-dimensional illumination intensity model to obtain a plurality of visible light reconstructed images includes: Decomposing any original visible light image among the plurality of original visible light images to obtain an original low-frequency sub-band and at least one high-frequency sub-band; Performing illumination compensation on the original low-frequency sub-band based on the three-dimensional illumination intensity model to obtain a first low-frequency sub-band; performing local contrast enhancement on the first low-frequency sub-band to obtain a second low-frequency sub-band; The at least one high-frequency sub-band and the second low-frequency sub-band are reconstructed to obtain the visible light reconstructed image.

6. The autonomous flight inspection method of a UAV according to claim 3, characterized in that: The stitching process of the plurality of visible light enhanced images to obtain the panoramic visible light image includes: performing feature detection on a first visible light-enhanced image and a second visible light-enhanced image among the multiple visible light-enhanced images to obtain a first feature point set of the first visible light-enhanced image and a second feature point set of the second visible light-enhanced image, where the first visible light-enhanced image and the second visible light-enhanced image are respectively obtained based on two continuously acquired original visible light images; Constructing a geometric constraint model based on the shooting parameters of the plurality of visible light enhanced images; matching feature points in the first feature point set and the second feature point set using the geometric constraint model to obtain a plurality of first feature point pairs; Obtaining image change parameters based on the multiple groups of first feature point pairs; Affine transformation and fusion are performed on the multiple visible light enhanced images based on the image change parameters to obtain a panoramic visible light image.

7. The autonomous flight inspection method of a UAV according to any one of claims 1 to 6, characterized in that: The adjusting of the inspection path and inspection parameters of the UAV inspection operation according to the equipment status detection report includes: Obtaining a fault area and a risk area in the target environment according to the equipment status monitoring report, and determining an inspection path of the drone according to the fault area and the risk area, wherein the inspection path passes through the fault area and the risk area; According to the equipment status monitoring report, equipment fault information in the target environment is obtained; according to the faulty equipment information, inspection parameters of the UAV are determined, and the inspection path includes imaging equipment parameters and flight parameters.

8. An autonomous flight inspection device for drones, characterized in that: include: An acquisition module is used to acquire multiple original visible light images, multiple infrared thermal imaging images, and multiple ultraviolet imaging images at multiple viewing angles in a target environment; a stitching module, configured to stitch the plurality of original visible light images to obtain a panoramic visible light image; A construction module, configured to construct a quantized energy field according to the infrared thermal imaging image and the ultraviolet imaging image; a mapping module, configured to perform correlation mapping between the quantized energy field and the panoramic visible light image to obtain a first fused image; a fusion module, configured to perform gradient processing on the plurality of infrared thermal imaging images and the plurality of ultraviolet imaging images, and adjust the pixel value distribution of the first fused image according to gradient information obtained by the gradient processing to obtain a second fused image; a color deviation compensation module, configured to convert the second fused image into a target color space, and perform color deviation compensation on the converted second fused image in the target color space to obtain a multimodal target image; A diagnosis module, configured to input the multimodal target image into a preconfigured fault diagnosis model to generate a device status detection report; The determination module is used to determine the inspection path and inspection parameters of the drone based on the equipment status detection report.

9. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and leads; Wherein, a computer program is stored on the memory, and the processor implements the steps of the autonomous flight inspection method of the UAV according to any one of claims 1 to 7 when executing the computer program on the memory; The processor, the memory and the communication interface are connected via wires to achieve data transmission.

10. A computer-readable storage medium, characterized in that The computer scale storage medium stores corresponding instructions, and the corresponding instructions can enable the computer to implement the steps of the autonomous flight inspection method of the unmanned aerial vehicle according to any one of claims 1 to 7.

Images