Unmanned aerial vehicle infrared visible light inspection data fusion display method

Abstract

The invention provides an unmanned aerial vehicle infrared visible light inspection data fusion display method, which comprises the following steps: acquiring visible light and infrared light image data shot by a dual-light camera of an unmanned aerial vehicle, uploading the locally stored image data to a cloud end for image registration fusion processing, quickly acquiring the image data, performing image registration by using an SURE operator, and displaying the image data. According to the method, infrared and visible light fusion is completed through multi-scale wavelet transformation after registration, a fusion result is subjected to wavelet inversion to form a final result, an image processing result is displayed through an image management system, and an image data management system based on cloud service is developed by adopting an SSM framework and a SpringBoot framework. According to the unmanned aerial vehicle infrared visible light inspection data fusion display method, the visible light image data and the infrared light image data shot by the unmanned aerial vehicle can be rapidly fused, and the fused view is clearly displayed to the user.

Description

Technical Field

[0001] This application belongs to the technical field of infrared and visible light fusion optics, and particularly relates to a method for fusing and displaying infrared and visible light inspection data of an unmanned aerial vehicle (UAV). Background Art

[0002] In recent years, to meet the demand for electricity in national economic and social development, new requirements have been put forward for the stability of power grid equipment operation. At the same time, with the digital transformation and upgrading of the power grid system, artificial intelligence technology has gradually been applied in the inspection work of power grid equipment. As the front-end technology of artificial intelligence, intelligent perception aims to detect environmental information in the external space using a variety of sensing devices, which can provide an information basis for the subsequent intelligent decision-making tasks and help achieve end-to-end artificial intelligence applications. Among them, vision is the most intuitive and important perception way, and the digital images obtained through imaging sensing devices are important carriers and manifestations of visual information.

[0003] However, the existing image fusion and display methods have a slow fusion and display speed for the visible light and infrared light image data captured by UAVs, and cannot clearly display the fused view to users in a timely manner. Therefore, a method for fusing and displaying infrared and visible light inspection data of UAVs is specifically proposed. Summary of the Invention

[0004] The embodiment of this application provides a method for fusing and displaying infrared and visible light inspection data of UAVs, which can quickly fuse the visible light and infrared light image data captured by UAVs and clearly display the fused view to users, solving the problem that the existing image fusion and display methods have a slow fusion and display speed for the visible light and infrared light image data captured by UAVs and cannot clearly display the fused view to users in a timely manner.

[0005] The embodiment of this application provides a method for fusing and displaying infrared and visible light inspection data of UAVs, including:

[0006] Collect the visible light and infrared light image data captured by the dual-light camera of the UAV and upload it to the cloud;

[0007] Fuse and locate the visible light and infrared light images in the cloud to obtain the fused image data;

[0008] Store the fused image data in the cloud;

[0009] Display the fused image data through an image management system.

[0010] In a feasible implementation manner, the uploading to the cloud is specifically:

[0011] After collecting the image data, upload the image data saved locally to the cloud for image registration and fusion processing;

[0012] Then, the processing result is fed back to this machine, and a connection between the local and the cloud is established for data transmission;

[0013] The data connection method between the local and the cloud is as follows: before using the Alibaba Cloud direct upload service, open the OSS service and create a Bucket. After opening the OSS service, create a Bucket in the management console to obtain a storage space; among them, the OSS is object storage, and the Bucket is an object storage space.

[0014] In a feasible implementation manner, the visible light and infrared light image data captured by the collection UAV's dual - light camera are registered using the SURE operator.

[0015] In a feasible implementation manner, the fusion and positioning of the visible light and infrared light images in the cloud are completed through multi - scale wavelet transform, specifically including:

[0016] Perform wavelet transform on the visible light image and the infrared visible light image to respectively form low - frequency components and high - frequency components;

[0017] Fuse the low - frequency components with the low - frequency components and the high - frequency components with the high - frequency components. After wavelet inverse transform of the fusion result, a final result is formed to obtain the fused image data.

[0018] In a feasible implementation manner, the multi - scale wavelet transform is implemented through the Meyer function. The Meyer wavelet α and the scaling function β are defined in the frequency domain:

[0019]

[0020] Among them, v(x) is an auxiliary function for constructing the Meyer wavelet, and the calculation is as follows:

[0021]

[0022] According to the decomposed low - frequency data part, select the scale coefficient, synthesize the contour of the image to be fused, clearly display the fused image, and enhance the visual effect of the fused image.

[0023] In a feasible implementation manner, the image data management system based on cloud services is developed using the SSM framework and the Spring Boot framework, and is mainly divided into 4 layers, namely the presentation layer, the control layer, the business layer, and the persistence layer;

[0024] The persistence layer uses the Mybatis framework to perform database access - related operations;

[0025] The specific process for a user to send a request through the presentation - layer page to obtain the processing result is as follows:

[0026] The binding method for the event triggered by the user clicking on the page sends an object request through the browser; the method bound to this event will send the request to SpringMVC, and the front controller DispatcherServlet in SpringMVC intercepts this request;

[0027] After DispatcherServlet intercepts the request, it calls the handler mapper. The handler mapper finds the specific controller method based on the URL, and HandlerAdapter forwards this request to the corresponding controller method;

[0028] The corresponding controller receives the request, encapsulates the data with VO, and calls the corresponding business layer method through Spring IoC to process this request. The business layer is used to specifically process the business request. When the business layer needs to perform data processing, it can use Spring IoC to call the corresponding Repository interface. The database operation-related methods have been pre-encapsulated in the MyBatis framework. After the database operation is completed, the business layer executes the business process operation and returns the processed result to the control layer. The control layer creates a view resolver and returns a suitable view to the front controller after parsing;

[0029] Finally, the view is rendered through the Bootstrap framework and displayed to the user in the browser.

[0030] A method for fusing and displaying drone infrared and visible light inspection data provided by an embodiment of this application. After technicians collect the visible light and infrared light image data captured by the drone's dual-light camera, they upload the image data saved locally to the cloud for image registration and fusion processing. The image data is quickly collected and image registration is performed using the SURE operator. After registration, the infrared and visible light are fused through multi-scale wavelet transform. The principle of image fusion is to perform wavelet transform on the infrared and visible light images to form low-frequency components and high-frequency components. During the image fusion process, the low-frequency components are fused with the low-frequency components, and the high-frequency components are fused with the high-frequency components. The fusion result forms the final result after wavelet inverse transformation. The display of the image processing result is shown through an image management system. The cloud service-based image data management system is developed using the SSM framework and the Spring Boot framework, and can be mainly divided into 4 layers, namely the presentation layer, the control layer, the business layer, and the persistence layer. The view is rendered through the Bootstrap framework and displayed to the user in the browser. This method for fusing and displaying drone infrared and visible light inspection data can quickly fuse the visible light and infrared light image data captured by the drone and clearly display the fused view to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic structural diagram of the method for fusing and displaying drone infrared and visible light inspection data provided by this application;

[0032] Figure 2 It is a schematic diagram of the principle of image fusion. Specific implementation manners

[0033] In order to enable those skilled in the art to better understand the technical solutions in this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0034] The unmanned aerial vehicle (UAV) can take infrared images and visible light images by carrying an infrared camera and a visible light camera, so as to obtain more comprehensive information under different lighting conditions. For example, in search and rescue missions, the UAV can use the infrared camera to search for missing persons at night or in low light conditions, and then use the visible light camera to obtain more detailed images during the day or in high light conditions.

[0035] The infrared and visible light inspection data fusion display method is to fuse infrared images and visible light images to obtain more comprehensive and accurate information. However, the existing image fusion display methods have a slow fusion speed for the visible light and infrared light image data captured by the UAV, and the quality after fusion is poor, and the fused view cannot be clearly displayed to the user in a timely manner.

[0036] This application uses the SURE operator for image registration. After registration, the infrared and visible light are fused through multi-scale wavelet transform. The image processing results are displayed through an image management system. The image data management system based on cloud service is developed using the SSM framework and the Spring Boot framework, and the view can be rendered through the Bootstrap framework and displayed to the user in the browser, so as to realize the fast fusion display of visible light and infrared light image data, ensure the fusion quality, and quickly and clearly display the fused view to the user.

[0037] The following will detail the specific structure of the UAV infrared and visible light inspection data fusion display method provided by this application with reference to the accompanying drawings.

[0038] Referring to Figure 1 As shown, the embodiment of this application provides a UAV infrared and visible light inspection data fusion display method, including:

[0039] Rapid acquisition and upload of visible light and infrared light image data;

[0040] Fusion positioning of visible light and infrared light images;

[0041] The image data is well preserved;

[0042] The result of image processing is displayed.

[0043] In one embodiment, the rapid acquisition of image data uses the SURE operator for image registration. After registration, the fusion of infrared and visible light is completed through multi-scale wavelet transform;

[0044] The SURE (Sum of Unsigned Residual Error) operator is an algorithm for image registration. It achieves registration by minimizing the residual between two images. The SURE operator is usually used in fields such as medical image registration.

[0045] The steps of using the SURE operator for image registration include:

[0046] 1. Define the image similarity metric: Select an appropriate image similarity metric function to compare the similarity between two images. Common similarity metric functions include the mean square error (MSE), cross-correlation coefficient (CC), etc.

[0047] 2. Calculate the SURE operator: Calculate the SURE operator according to the defined image similarity metric function. The SURE operator is the sum of the absolute values of the residuals of each pixel.

[0048] 3. Optimize the SURE operator: Use optimization algorithms (such as the gradient descent method, Newton's method, etc.) to minimize the SURE operator, thereby finding the optimal registration parameters.

[0049] 4. Iteratively update the registration parameters: In each iteration, update the registration parameters according to the results of the optimization algorithm, then recalculate the SURE operator, and continue the optimization until the SURE operator reaches the minimum value or meets the stopping condition.

[0050] 5. Apply the registration parameters: Apply the optimal registration parameters to the source image and the target image to achieve image registration.

[0051] It should be noted that the SURE operator needs to compare the two images pixel by pixel during the calculation process, so the computational complexity is relatively large. In practical applications, some acceleration techniques (such as parallel computing, image block division, etc.) are usually adopted to improve the computational efficiency.

[0052] Multi-scale wavelet transform is an image processing technology that decomposes an image into sub-bands of different scales and directions for feature extraction, image compression, denoising, etc.

[0053] The basic idea of the multi-scale wavelet transform is to filter the image using wavelet functions and decompose the image into sub-bands of different scales and directions. The wavelet function is a localized function that has locality at a certain scale and can capture the details and textures in the image.

[0054] In the multi-scale wavelet transform, the discrete wavelet transform (DWT) is usually used to decompose the image. The DWT decomposes the image into a series of sub-bands, and each sub-band represents the features of the image at different scales and directions. These sub-bands can be reconstructed through the inverse discrete wavelet transform (IDWT) to restore the original image.

[0055] The multi-scale wavelet transform has many applications in image processing, such as image compression, denoising, image enhancement, feature extraction, etc. It can effectively capture the detail and texture information in the image while retaining the overall structure and shape of the image.

[0056] In one embodiment, in this system, after the technician has collected the image data, the image data saved locally needs to be uploaded to the cloud for image registration and fusion processing, and then the processing result is fed back to the local machine. A connection between the local and the cloud needs to be established for data transfer.

[0057] The data connection method between the local and the cloud is that before using the Alibaba Cloud direct upload service, the OSS service needs to be enabled and a Bucket needs to be created. After enabling the OSS service, a Bucket can be created in the management console. After completion, this storage space can be seen.

[0058] OSS is object storage, and Bucket is the object storage space.

[0059] The steps to enable the OSS (Object Storage Service) service and create a Bucket include:

[0060] 1. Log in to the console of the selected cloud service provider.

[0061] 2. Navigate to the Object Storage Service (OSS) or a similar storage service page.

[0062] 3. On the OSS service page, find the option of "Create Bucket" or "New Bucket".

[0063] 4. Enter the name of the Bucket, select the storage region (if there are multiple available regions), and set other attributes as needed.

[0064] 5. Click the "Create" or "OK" button to complete the creation of the Bucket.

[0065] After creating a Bucket, the information of the storage space can usually be seen in the console, including the Bucket name, storage region, capacity usage, etc. File upload, download, deletion, etc. operations can also be performed through the console or the corresponding API.

[0066] In one embodiment, the multi-scale wavelet transform is implemented by the Meyer function, and the Meyer wavelet α and the scaling function β are defined in the frequency domain:

[0067]

[0068] where v(x) is an auxiliary function for constructing the Meyer wavelet and is calculated as follows:

[0069]

[0070] According to the decomposed low-frequency data part, scale coefficients are selected to synthesize the contour of the image to be fused, clearly display the fused image, and enhance the visual effect of the fused image.

[0071] In one embodiment, as Figure 2 shown, the principle of image fusion lies in performing wavelet transform on the infrared visible light image A and the visible light image B to form low-frequency components and high-frequency components. During the image fusion process, the low-frequency components are fused with the low-frequency components, and the high-frequency components are fused with the high-frequency components. The fusion result forms the final result after wavelet inverse transformation.

[0072] The display of the image processing result is through the image management system. The image data management system based on cloud service is developed using the SSM framework and the Spring Boot framework, and can be mainly divided into 4 layers, namely the presentation layer, the control layer, the business layer, and the persistence layer;

[0073] 1. Presentation Layer: It is the user interface part of the system, responsible for interacting with users, displaying data, and accepting user input. It usually includes user interfaces, web pages, application program interfaces, etc.

[0074] 2. Controller Layer: It is responsible for processing user requests and operations, and coordinating the communication between the presentation layer and the business layer. It receives requests from the presentation layer, parses the request parameters, calls the corresponding business logic, and returns the result to the presentation layer.

[0075] 3. Business Layer: It is the core part of the system, responsible for implementing business logics and rules. It processes and executes specific business operations, and performs calculations, validations, and processing on data.

[0076] 4. Persistence Layer: Responsible for interacting with the database or other persistent storage and managing data persistence operations. It is responsible for storing data in the database and retrieving data from the database.

[0077] By dividing the system into different layers, each layer responsible for specific functions, the maintainability, scalability, and reusability of the code can be improved. This layered architecture pattern helps in the organization of the code and the division of labor among developers. Different layers can be responsible for different development teams or developers, thus improving development efficiency and quality.

[0078] The persistence layer uses the Mybatis framework to perform database access-related operations;

[0079] The specific process for the user to send a request through the presentation layer page and obtain the processing result is as follows:

[0080] The user clicks on the page to trigger the binding method of the event, and sends an object request through the browser. The binding method of this event will send the request to SpringMVC, and the front-end controller DispatcherServlet in SpringMVC intercepts this request;

[0081] After DispatcherServlet intercepts the request, it calls the handler mapper. The handler mapper finds the specific controller method based on the URL, and HandlerAdapter forwards this request to the corresponding controller method. The corresponding controller receives the request, encapsulates the data with VO, and calls the corresponding business layer method through Spring IoC to process this request. The business layer is used to perform specific processing of business requests. When the business layer needs to perform data processing, it can use Spring IoC to call the corresponding Repository interface. The database operation-related methods have been pre-encapsulated in the MyBatis framework. After the database operation is completed, the business layer executes the business process operation, and returns the processed result to the control layer. The control layer creates a view resolver, and after parsing, returns a suitable view to the front-end controller;

[0082] After the above steps are completed, the view can be rendered through the Bootstrap framework and displayed to the user in the browser.

[0083] The method for fusing and displaying the inspection data of visible light and infrared light of an unmanned aerial vehicle provided by this application. After technicians collect the visible light and infrared light image data captured by the dual-light camera of the unmanned aerial vehicle, they upload the image data saved locally to the cloud for image registration and fusion processing. The SURE operator is used for image registration in the rapid acquisition of image data. After registration, the fusion of infrared and visible light is completed through multi-scale wavelet transform. The principle of image fusion lies in performing wavelet transform on the infrared and visible light images to form low-frequency components and high-frequency components. During the image fusion process, the low-frequency components are fused with the low-frequency components, and the high-frequency components are fused with the high-frequency components. The fusion result forms the final result after wavelet inverse transformation. The display of the image processing result is shown through an image management system. The image data management system based on cloud services is developed using the SSM framework and the Spring Boot framework, and can be mainly divided into 4 layers, namely the presentation layer, the control layer, the business layer, and the persistence layer. The view is rendered through the Bootstrap framework and displayed to the user in the browser. This method for fusing and displaying the inspection data of visible light and infrared light of an unmanned aerial vehicle can quickly fuse the visible light and infrared light image data captured by the unmanned aerial vehicle and clearly display the fused view to the user. It solves the problems of the existing image fusion and display methods, such as the slow fusion speed of the visible light and infrared light image data captured by the unmanned aerial vehicle, the poor quality after fusion, and the inability to timely and clearly display the fused view to the user.

[0084] It is easy to understand that those skilled in the art can combine, split, reorganize, etc. the embodiments of this application based on several embodiments provided by this application to obtain other embodiments, and these embodiments do not exceed the protection scope of this application.

[0085] The above specific implementation manners further elaborate on the purpose, technical solutions, and beneficial effects of the embodiments of this application. It should be understood that the above are only the specific implementation manners of the embodiments of this application and are not used to limit the protection scope of the embodiments of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of this application should be included in the protection scope of the embodiments of this application.

Claims

1. An infrared and visible light inspection data fusion display method for an unmanned aerial vehicle, characterized in that: including; Collect visible light and infrared light image data captured by the dual - light camera of the UAV and upload it to the cloud; Fuse and locate the visible light and infrared light images in the cloud to obtain the fused image data; Store the fused image data in the cloud; Display the fused image data through the image management system.

2. The method for fusing and displaying UAV infrared - visible light inspection data according to claim 1, wherein: The uploading to the cloud is specifically as follows: After collecting the image data, upload the image data saved locally to the cloud for image registration and fusion processing; Then feedback the processing result to the local machine to establish a connection between the local and the cloud for data transfer; The data connection method between the local and the cloud is: Before using the Alibaba Cloud direct upload service, open the OSS service and create a Bucket. After opening the OSS service, create a Bucket in the management console to obtain a storage space; where the OSS is object storage and the Bucket is an object storage space.

3. The method for fusing and displaying UAV infrared - visible light inspection data according to claim 1, wherein: The collection of visible light and infrared light image data captured by the dual - light camera of the UAV uses the SURE operator for image registration.

4. The method for fusing and displaying UAV infrared - visible light inspection data according to claim 1, wherein: The fusion and location of the visible light and infrared light images in the cloud are completed through multi - scale wavelet transform, specifically including; Perform wavelet transform on the visible light image and the infrared - visible light image to respectively form low - frequency components and high - frequency components; Fuse the low - frequency components with the low - frequency components and the high - frequency components with the high - frequency components. After wavelet inverse transformation of the fusion result, form the final result to obtain the fused image data.

5. The method for fusing and displaying UAV infrared - visible light inspection data according to claim 4, wherein: The multi - scale wavelet transform is implemented through the Meyer function. The Meyer wavelet α and the scaling function β are defined in the frequency domain: where v(x) is an auxiliary function for constructing the Meyer wavelet, and the calculation is as follows: According to the decomposed low - frequency data part, select the scale coefficient, perform the synthesis of the contours of the images to be fused, clearly display the fused image, and enhance the visual effect of the fused image.

6. The method for fusing and displaying UAV infrared - visible light inspection data according to claim 1, wherein: The image data management system based on cloud service is developed using the SSM framework and the Spring Boot framework, and can be mainly divided into four layers, namely the presentation layer, the control layer, the business layer, and the persistence layer; The persistence layer uses the Mybatis framework to perform database access - related operations; The specific process for the user to send a request from the presentation - layer page to obtain the processing result is as follows: The user clicks the page to trigger the binding method of the event and sends an object request through the browser; the method bound to this event will send the request to Spring MVC, and the front - end controller DispatcherServlet in Spring MVC intercepts this request; After the DispatcherServlet intercepts the request, it calls the handler mapper. The handler mapper finds the specific controller method based on the URL, and the HandlerAdapter forwards the request to the corresponding controller; The corresponding controller receives the request, encapsulates the data with VO, and calls the corresponding business layer method through Spring IoC to process the request. The business layer is used to specifically handle business requests. When the business layer needs to perform data processing, it uses Spring IoC to call the corresponding Repository interface. The database operation-related methods are encapsulated in the MyBatis framework. After the database operation is completed, the business layer executes the business process operation and returns the processed result to the control layer. The control layer creates a view resolver, and after parsing, returns a suitable view to the front controller; The view is rendered through the Bootstrap framework and displayed in the browser.

Images