Image fusion defect detection method and device and readable storage medium

Abstract

The invention discloses an image fusion defect detection method and device and a readable storage medium, and the method comprises the steps: collecting the multi-source image data and laser radar data of a power transformation main device through a plurality of sensors, and enabling the multi-source image data to comprise the defect information of the power transformation main device; preprocessing the multi-source image data, and enhancing the defect information; fusing the preprocessed multi-source image data into a multi-source fused image through an image fusion algorithm, wherein the multi-source fused image is obtained by overlapping different images in the multi-source image data based on weights; analyzing the multi-source fusion image through a deep learning model to obtain defect information; and generating a three-dimensional model of the power transformation main equipment based on the multi-source image data and the laser radar data, and superposing the defect information on the three-dimensional model based on augmented reality (AR). The defect detection equipment can comprehensively detect the power transformation main equipment from different dimensions, and it is ensured that no potential defect is missed.

Description

Technical Field

[0001] This application relates to the field of image recognition, and particularly to a method, apparatus, and readable storage medium for defect detection in image fusion. Background Art

[0002] With the continuous expansion of the scale of the power system, the safe operation of substation equipment is of crucial importance. To ensure the reliability of the power grid, in the prior art, manual inspections, infrared thermal imaging detections, visible light image analyses, etc. are generally used to detect defects in main substation equipment. These methods usually rely on a single type of sensor or simple image analysis algorithms. For example, the appearance information of the equipment is obtained through a visible light camera, the temperature anomaly is monitored using an infrared thermal imager, or the external state of the equipment is regularly inspected manually.

[0003] Currently, in the technical solutions for detecting main substation equipment, there are significant limitations in the detection accuracy and accuracy of equipment defects, making it difficult to meet the increasingly complex detection requirements of modern substation equipment. On the one hand, a single sensor is limited by its own acquisition capabilities and is difficult to comprehensively capture various defect information of the equipment; on the other hand, due to factors such as environmental light, weather changes, and the complexity of the equipment surface, existing detection means are prone to misjudgment or missed detections. In addition, traditional detection methods have a low recognition accuracy when faced with minor or hidden defects in the equipment, which may lead to the failure to detect equipment failures in a timely manner.

[0004] Therefore, how to efficiently and accurately identify equipment defects in main substation equipment has become an urgent problem to be solved. Summary of the Invention

[0005] Embodiments of this application provide a method, apparatus, and readable storage medium for defect detection in image fusion, which can comprehensively detect main substation equipment from different dimensions through the combination of data collected by multiple sensors to ensure that no potential defects are missed. The technical solution is as follows:

[0006] In a first aspect, a method for defect detection in image fusion is provided, which is applied to a defect detection device for detecting defects in main substation equipment. The method includes: collecting multi-source image data and lidar data of the main substation equipment through multiple sensors, where the multi-source image data has defect information of the main substation equipment; preprocessing the multi-source image data to enhance the defect information; fusing the preprocessed multi-source image data into a multi-source fusion image through an image fusion algorithm, where the multi-source fusion image is obtained by weighted superposition of different images in the multi-source image data; analyzing the multi-source fusion image through a deep learning model to obtain the defect information; generating a three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimposing the defect information on the three-dimensional model based on augmented reality (AR).

[0007] In combination with the first aspect, the multiple sensors include a high-definition camera, an infrared thermal imager, a depth camera, and a lidar, and the multi-source image data includes visible light images, infrared thermal imaging images, and depth images.

[0008] In combination with the first aspect, in some embodiments of the first aspect, preprocessing the multi-source image data to enhance the defect information includes: denoising the multi-source image data, image alignment, color correction, and image enhancement to enhance the defect information; wherein, the denoising process is used to remove noise in the multi-source image data and improve the clarity of the multi-source image data; the image alignment is used to place the multi-source image data in the same coordinate system; the color correction is used to ensure that the colors of the multi-source image data from different sources are consistent; and the image enhancement is used to highlight the details on the surface of the main substation equipment in the multi-source image data.

[0009] In combination with the first aspect, in some embodiments of the first aspect, the image fusion algorithm includes determining the weights of images from different sources based on image quality, sensor characteristics, and defect features, and fusing the images from different sources according to the weights to generate a fused image.

[0010] In combination with the first aspect, in some embodiments of the first aspect, the deep learning model includes a convolutional neural network (CNN). The deep learning model is used to determine the defect type, defect location, and defect severity of the main substation equipment, and the defect types include cracks, corrosion, overheating, and surface damage.

[0011] In combination with the first aspect, in some embodiments of the first aspect, the deep learning model includes an image defect quality assessment algorithm, and the image defect quality assessment algorithm includes the following steps:

[0012] Step (1). Extract multiple image features from the detected image defect area. The image features include the area of the image defect area, the sharpness of the edge of the image defect area, the regularity of the shape of the image defect area, and the surface texture of the image defect area.

[0013] Step (2). Based on the extracted image features, calculate a severity score in combination with the geometric attributes and surface features of the image defect area. The severity score is used to divide the severity level of the image defect area.

[0014] Step (3). By comparing with a standard equipment model, exclude false detections caused by factors such as lighting, shadows, and noise, and determine the image defect area of the real damage.

[0015] Step (4). Based on the severity score, sort the image defect areas of the real damage in descending order, and sequentially assign repair priorities from high to low, where the repair priority is used to indicate the processing order of the image defect areas.

[0016] Combined with the first aspect, in some embodiments of the first aspect, the three-dimensional model includes the geometric shape and surface features indicating the main substation equipment, the defect information includes the specific spatial position of the defect on the three-dimensional model, after generating the three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimposing the defect information on the three-dimensional model based on augmented reality (AR), the method further includes:

[0017] Display the repair solutions and steps of the defects on the main substation equipment through the AR.

[0018] In a second aspect, there is provided a defect detection device for image fusion, including:

[0019] A data acquisition unit, configured to collect multi-source image data and lidar data of the main substation equipment through multiple sensors, where the multi-source image data has defect information of the main substation equipment;

[0020] An image preprocessing unit, configured to preprocess the multi-source image data to enhance the defect information;

[0021] An image data fusion unit, configured to fuse the preprocessed multi-source image data into a multi-source fusion image through an image fusion algorithm, where the multi-source fusion image is obtained by superimposing different images in the multi-source image data based on weights;

[0022] A defect recognition unit, configured to analyze the multi-source fusion image through a deep learning model to obtain the defect information;

[0023] A defect visualization unit, configured to generate a three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimpose the defect information on the three-dimensional model based on AR.

[0024] In a third aspect, there is provided a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the methods in the foregoing first aspect or any one of the embodiments of the first aspect are implemented.

[0025] In a fourth aspect, there is provided a computer program product, including computer program instructions, and when the computer program instructions run on a computer, the computer is caused to execute the methods in the foregoing first aspect or any one of the embodiments of the first aspect.

[0026] The above-mentioned defect detection method for image fusion has the following beneficial effects:

[0027] 1. Through the combination of multi-source sensor data, the defect detection device can comprehensively detect the device from different dimensions, enhancing the accurate detection ability for micro-defects, difficult-to-identify defects, and defects in complex environments, and improving the ability to identify potential defects.

[0028] 2. The defect identification and location using deep learning algorithms in this application enable the defect detection device to not only quickly and accurately identify various types of defects but also precisely calibrate the location and severity of the defects in the image. In this way, equipment maintenance personnel can quickly locate and handle equipment failures, reducing the time and error of manual detection, and thus reducing the downtime and improving work efficiency.

[0029] 3. The image fusion algorithm in this application can dynamically adjust the fusion strategy of different sensor images according to environmental changes (such as lighting, weather, etc.) and the characteristics of the device surface (such as surface texture, material differences, etc.), enabling clear capture of device defects regardless of low light, strong light, or complex backgrounds. In this way, this method can adapt to various complex environments and avoid detection errors of traditional detection methods under environmental changes.

[0030] 4. The three-dimensional model generated by lidar and depth images in this application enables the location, size, and severity of device defects to be more intuitively displayed. Based on the augmented reality (AR) technology, equipment maintenance personnel can view the three-dimensional structure and defect conditions of the device in real time through AR devices, improving the accuracy of defect detection and also enhancing the efficiency and accuracy of the repair process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a flowchart of a method for defect detection provided by an embodiment of this application;

[0032] Figure 2 It is a flowchart of a method for preprocessing multi-source image data provided by an embodiment of this application;

[0033] Figure 3 It is a flowchart of a method for fusing multi-source image data into multi-source fused images provided by an embodiment of this application;

[0034] Figure 4 It is a flowchart of a method for analyzing multi-source fused images to obtain defect information provided by an embodiment of this application;

[0035] Figure 5 It is a flowchart of a method for generating a three-dimensional model provided by an embodiment of this application;

[0036] Figure 6 It is a schematic diagram of a defect detection device for image fusion provided by an embodiment of this application;

[0037] Figure 7 Schematic diagram of the hardware structure of a defect detection device provided by an embodiment of the present application;

[0038] Figure 8 Internal structure diagram of a computer-readable storage medium provided by an embodiment of the present application. Detailed implementation manners

[0039] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The technical solutions in the embodiments of the present application will be clearly and elaborately described below with reference to the accompanying drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or". For example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0040] The statements such as "in the embodiments of the present application" described in the present application mean that a specific feature, structure or characteristic described in the embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" and the like that appear in different parts of the present application do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. In addition, the terms "comprise", "include", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0041] Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and cannot be construed as implying or suggesting relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0042] For the convenience of understanding, the following explanations are made for the related concepts involved in the embodiments of the present application first:

[0043] Augmented Reality (AR): It is a technology that superimposes virtual information (such as images, sounds, texts, 3D models, etc.) on the environment and presents it to the user in real time through a device. In the embodiments of the present application, the AR technology is used to superimpose the defects of the main substation equipment on the 3D model of the main substation equipment and display the detected defects to the user in real time.

[0044] The above are some of the related concepts involved in the embodiments of this application.

[0045] In order to solve the problem that the existing technology relies on a single type of sensor or a simple image analysis algorithm for defect detection, and the recognition accuracy is low when there are small defects or hidden defects in the equipment, the present application proposes an image fusion defect detection method, which includes: the defect detection device collects multi-source image data and laser radar data of the substation main equipment through multiple sensors, and the multi-source image data contains the defect information of the substation main equipment; pre-processes the multi-source image data to enhance the defect information; fuses the pre-processed multi-source image data into a multi-source fusion image through an image fusion algorithm, and the multi-source fusion image is obtained by superimposing different images in the multi-source image data based on weights; analyzes the multi-source fusion image through a deep learning model to obtain the defect information; generates a three-dimensional model of the substation main equipment based on the multi-source image data and the laser radar data, and superimposes the defect information on the three-dimensional model based on augmented reality AR. Among them, through the combination of data collected by multiple sensors, the defect detection equipment can comprehensively detect the substation main equipment from different dimensions to ensure that no potential defects are missed.

[0046] The following describes the image fusion defect detection method, device and readable storage medium involved in the embodiments of the present application through Examples 1 to 3. Among them, Example 1 is used to describe the image fusion defect detection method; Example 2 is used to describe the image fusion defect detection device; Example 3 is used to describe the image fusion defect detection readable storage medium.

[0047] Example 1

[0048] For example, Figure 1 A method flow chart of a defect detection method is shown, which specifically includes:

[0049] S101. Collect data based on multiple sensors.

[0050] In the embodiment of the present application, the defect detection device uses a variety of sensors to collect data from the main substation equipment at multiple angles and in multiple modes. The multiple sensors include, but are not limited to, high-definition cameras, infrared thermal imagers, depth cameras, and laser radars. The collected data include multi-source image data and laser radar data. The multi-source image data include visible light images, infrared thermal imaging images, and depth images. The multi-source image data contains defect information of the main substation equipment.

[0051] Exemplarily, multiple sensors in the defect detection device work simultaneously. Different types of sensors can capture different-dimensional information of the main power transformation equipment, providing a comprehensive information source. Specifically, a high-definition camera is used to capture visible light images on the surface of the main power transformation equipment to provide detailed information on the appearance of the main power transformation equipment; an infrared thermal imager is used to detect the surface temperature of the main power transformation equipment. Especially when the main power transformation equipment overheats due to a fault, the temperature on the surface of the main power transformation equipment has a significant anomaly; a depth camera and a lidar are used to obtain three-dimensional spatial data of the main power transformation equipment, and the three-dimensional spatial data is used to construct the shape and surface of the main power transformation equipment in the AR space.

[0052] It should be noted that this multi-source data acquisition method can obtain information on the main power transformation equipment from different dimensions and angles, avoiding the limitations of using only a single sensor. For example, by jointly using an infrared thermal imager and a high-definition camera, an infrared thermal image and a visible light image are obtained. Analyzing the combination of the infrared thermal image and the visible light image can effectively alleviate the adverse effects of different environmental conditions (such as low light or strong light) on the judgment of defects on the main power transformation equipment, improving the accuracy of defect detection on the main power transformation equipment. Another example is that by jointly using a lidar and a depth camera, lidar data and a depth image are obtained. Analyzing the combination of the lidar data and the depth image can obtain the three-dimensional structure and surface structure of the main power transformation equipment, thereby identifying and locating defects that cannot be captured by traditional imaging techniques.

[0053] S102. Preprocess the collected multi-source image data.

[0054] In the embodiment of the present application, the defect detection device preprocesses the multi-source image data collected in step S101. The preprocessing steps include denoising, image alignment, color correction, and image enhancement. Among them, denoising is used to remove noise in the multi-source image data and improve the clarity of the multi-source image data; image alignment is used to place the multi-source image data under the same coordinates; color correction is used to ensure that the colors of the multi-source image data from different sources are consistent; image enhancement is used to highlight the details on the surface of the main power transformation equipment in the multi-source image data.

[0055] It should be noted that preprocessing can significantly improve the quality of multi-source image data, especially when facing different sensors, because different transmitters are each set with different parameters, and the obtained multi-source image data has differences. Among them, image alignment and color correction can ensure the consistency of multi-source image data in spatial position and visual color; denoising and image enhancement can highlight the defect areas in the multi-source image data, improving the accuracy of the defect detection algorithm. Especially when facing tiny defects or blurred defects, the tiny defects or blurred defects in the multi-source image data are more obvious in the enhanced image.

[0056] S103. Fuse multi-source image data into a multi-source fused image.

[0057] In an embodiment of the present application, the image fusion algorithm includes determining the weights of images from different sources based on image quality, sensor characteristics, and defect characteristics, and fusing the images from different sources according to the weights to generate a fused image.

[0058] In an embodiment of the present application, the defect detection device fuses the preprocessed multi-source image data into a multi-source fused image through the image fusion algorithm. The multi-source fused image is obtained by superimposing different images in the multi-source image data based on weights and is used for subsequent defect detection. Among them, the defect detection device can adjust the weights of different images in each multi-source image data based on the characteristics of different sensors and the acquisition environment.

[0059] Specifically, the defect detection device aligns the spaces of different images in the multi-source image data through the image fusion algorithm, fuses different image data, and the fused multi-source fused image satisfies the complementarity of the image data of each sensor and maximizes the contribution of the image data. For example, an infrared thermal imager captures an infrared thermal image, which contains the temperature information of the main substation equipment, and a high-definition camera captures a visible light image, which contains the surface details of the main substation equipment. When the infrared thermal image and the visible light image are fused, based on the shooting quality of these two image data and the state of the sensor, the most valuable image data is preferentially selected to form a clear and information-rich multi-source fused image.

[0060] It should be noted that through dynamic adjustment based on the image fusion algorithm, the defect detection device can adapt to different acquisition environments and automatically optimize the quality of image fusion. Especially for complex scenarios, such as scenarios with large light changes or complex environmental backgrounds, the defect detection device can obtain the optimal image data fusion, while complementing the image data collected by different sensors and reducing the problems of image distortion or information loss caused by a single sensor.

[0061] S104. Analyze the multi-source fused image to obtain defect information.

[0062] In an embodiment of the present application, the defect detection device analyzes the multi-source fused image through a deep learning model to obtain defect information. Among them, the deep learning model includes a convolutional neural network CNN. The deep learning model is used to determine the defect type, defect location, and defect severity of the main substation equipment. The defect type includes but is not limited to cracks, corrosion, overheating, and surface damage.

[0063] Specifically, the defect detection device uses a deep learning model to analyze multi-source fusion images. Among them, the deep learning model can adopt the training method of convolutional neural network CNN to learn and identify various defects on the surface of the main substation equipment from a large amount of training data, such as cracks, corrosion, scratches, etc. The deep learning model obtains the defects in the multi-source fusion image by extracting multi-layer features, including the type of the defect and the position and size of the defect in the multi-source fusion image. At the same time, the deep learning model can adapt to the complex geometric shapes and various environmental conditions on the surface of the main substation equipment and has the ability to accurately identify tiny defects.

[0064] It should be noted that the defect detection of the main substation equipment by the deep learning model no longer relies on artificial rules or simple image processing methods, but detects the defects of the main substation equipment by learning the defect features in various environments. This method improves the accuracy and intelligence level of defect detection. Especially when dealing with tiny defects or defects with complex shapes that are difficult to identify by traditional methods, this method can quickly and accurately obtain these defects. At the same time, the deep learning model can also obtain the defect position in the main substation equipment, and this defect position is used to indicate the equipment maintenance personnel to quickly know the defect and reduce the time spent on maintenance work.

[0065] S105. Generate a 3D model and superimpose the defect information.

[0066] In the embodiment of the present application, the defect detection device generates a 3D model of the main substation equipment based on multi-source image data and lidar data, and superimposes the defect information on the 3D model based on AR technology to provide intuitive defect display and repair guidance for the equipment maintenance personnel.

[0067] Specifically, the defect detection device generates a 3D model of the main substation equipment based on lidar data and depth images. The 3D model is used to indicate the geometric shape of the main substation equipment and provide specific spatial information of the main substation equipment for the equipment maintenance personnel. At the same time, based on AR technology, the defect detection device superimposes the defect information of the main substation equipment into the 3D model. The equipment maintenance personnel can view the 3D model of the main substation equipment through the AR device to obtain the defects on the surface of the main substation equipment, as well as the spatial position, size and repair method of the defect.

[0068] In some embodiments, AR technology is used to support the equipment maintenance personnel to remotely diagnose the main substation equipment and repair the defects of the main substation equipment with remote assistance. Among them, the defect detection device also includes a repair plan and steps for displaying the defects on the main substation equipment through AR technology.

[0069] It should be noted that the 3D model of the main substation equipment can display the defects of the equipment more intuitively and accurately compared with the traditional 2D images, and can also help equipment maintenance personnel understand the structure and defect location of the main substation equipment. The defect detection equipment realizes precise defect visualization through AR technology. Equipment maintenance personnel can view the location and size of the defects of the main substation equipment in the AR interface, so as to quickly locate the defect area of the main substation equipment and repair it. Through AR technology, the repair operation of the defects of the main substation equipment by equipment maintenance personnel can be more intuitive and efficient, avoiding the problems of incorrect repair and missed repair in the traditional method.

[0070] In the embodiment of the present application, the defect detection equipment can also judge the severity of the defect by quantitatively evaluating the identified defect.

[0071] Specifically, the defect detection equipment extracts multiple features from the defect area, including the size, shape, edge sharpness, texture change, etc. of the defect, and calculates the weights of these features through a preset evaluation model to obtain the severity score of the defect area. The defect detection equipment sorts the defect areas based on the severity score to determine the priority of repairing the defect areas.

[0072] In some embodiments, the defect detection equipment can also compare the 3D model with the superimposed defect information with the standard equipment model to verify the authenticity of the defect, thus avoiding misjudgment or missed judgment. The standard equipment model is the 3D model of the main substation equipment without defect areas.

[0073] Exemplarily, based on Figure 1 the shown step S102, Figure 2 shows a flowchart of a method for preprocessing multi-source image data, which specifically includes:

[0074] S201. Denoising processing.

[0075] In the embodiment of the present application, the denoising processing can eliminate the environmental noise in the multi-source image data, enhance the defect information, and improve the quality of the multi-source image data. Among them, the defect detection equipment uses a filtering algorithm to remove the noise in the image and improve the clarity of the image.

[0076] S202. Image alignment.

[0077] In the embodiment of the present application, the image alignment is used to align the multi-source image data from different sensors to the same coordinate to ensure the consistency of the spatial information of each multi-source image data. Among them, the defect detection equipment performs spatial alignment on the images collected by different sensors so that they are in the same coordinate for subsequent fusion.

[0078] S203. Color correction.

[0079] In the embodiments of the present application, color correction is used to eliminate color differences caused by different sensors or lighting conditions, ensuring the color consistency of multi-source image data from different sources after fusion. Among them, the defect detection device performs color correction on the images collected by different sensors to ensure that the colors of the images from different sources are consistent.

[0080] S204. Image enhancement.

[0081] In the embodiments of the present application, image enhancement is used to enhance the contrast and sharpen the details of multi-source image data, highlighting the details on the surface of the main substation equipment in the multi-source image data, thereby highlighting possible defect areas and assisting subsequent defect detection. Among them, the defect detection device uses contrast enhancement and edge sharpening image enhancement methods to highlight the details on the surface of the main substation equipment and improve the detection accuracy of defects.

[0082] Exemplarily, based on Figure 1 the S103 step shown, Figure 3 the following shows a flowchart of a method for fusing multi-source image data into multi-source fused images, which specifically includes:

[0083] S301. Adjust the weights of image sources.

[0084] In the embodiments of the present application, when fusing multi-source image data, the defect detection device will automatically adjust the weights of each image source according to the characteristics of different sensors and the acquisition environment. For a specific description of the defect detection device adjusting the weights of image sources, reference can be made to the S103 step described above Figure 1 and will not be elaborated here.

[0085] S302. Optimize the quality of image fusion through an image fusion algorithm.

[0086] In the embodiments of the present application, for a specific description of the defect detection device dynamically adjusting through an image fusion algorithm to optimize the quality of image fusion, reference can be made to the S103 step described above Figure 1 and will not be elaborated here.

[0087] Exemplarily, based on Figure 1 the S104 step shown, Figure 4 the following shows a flowchart of a method for analyzing multi-source fused images to obtain defect information, which specifically includes:

[0088] S401. Extract multiple image features.

[0089] In an embodiment of the present application, the deep learning model includes an image defect quality assessment algorithm. Based on this image defect quality assessment algorithm, the deep learning model in the defect detection device extracts multiple image features from the detected image defect area. The image features include the area of the image defect area, the sharpness of the edge of the image defect area, the regularity of the shape of the image defect area, and the surface texture of the image defect area.

[0090] It should be noted that the features such as the area of the above image defect area, the sharpness of the edge of the image defect area, the regularity of the shape of the image defect area, and the surface texture of the image defect area are the basic attributes for describing the defect. These basic attributes of the defect are important data for subsequent defect assessment.

[0091] S402. Calculate the severity of the image defect.

[0092] In an embodiment of the present application, the deep learning model calculates a severity score by combining the extracted image features with the geometric attributes and surface features of the image defect area. The severity score is used to divide the severity level of the image defect area.

[0093] It should be noted that the severity score comprehensively considers factors such as the area of the defect, the symmetry of the defect shape, and the edge clarity, etc., and is used for quantitative evaluation of the defect, so as to divide the defect into multiple severity levels.

[0094] S403. Judge the authenticity of the image defect.

[0095] In an embodiment of the present application, the deep learning model excludes false detections caused by factors such as light, shadow, and noise by comparing with a standard device model, and determines the image defect area of the real damage.

[0096] It should be noted that the deep learning model compares and analyzes the shape of the defect area with the shape of the corresponding area in the standard device model to determine whether the detected defect is an actual defect.

[0097] S404. Sort the image defects.

[0098] In an embodiment of the present application, the deep learning model sorts the image defect areas of real damage based on the severity score in descending order, and assigns repair priorities from high to low in turn. The repair priority is used to indicate the processing order of the image defect area.

[0099] It should be noted that the equipment maintenance personnel can determine the processing order in the maintenance work according to the repair priority order of the image defect area, optimize the maintenance work process and improve the efficiency.

[0100] Exemplarily, based on Figure 1The step S105 shown Figure 5 shows a flowchart of a method for analyzing multi-source fusion images to obtain defect information, specifically including:

[0101] S501. Generate a 3D model.

[0102] In the embodiment of the present application, the defect detection device generates a 3D model of the main substation equipment based on multi-source image data and lidar data. The multi-source image data is a depth image, and the 3D model includes the geometric shape and surface features indicating the main substation equipment.

[0103] S502. Overlay defect information.

[0104] In the embodiment of the present application, the defect detection device overlays the defect information on the 3D model based on AR technology. The 3D model with the overlaid defect information is used to indicate the spatial position of the defect that the equipment maintenance personnel can see through the AR device. The defect information includes the specific spatial position of the defect on the 3D model.

[0105] S503. Provide repair guidance.

[0106] In the embodiment of the present application, the defect detection device displays the repair scheme and steps of the defect on the main substation equipment through AR technology to help the equipment maintenance personnel accurately perform the repair operation.

[0107] Embodiment 2

[0108] Figure 6 shows a schematic diagram of a defect detection device 60 for image fusion provided by the embodiment of the present application, including the following units:

[0109] The data acquisition unit 61 is used to collect multi-source image data and lidar data of the main substation equipment through multiple sensors. The multi-source image data has defect information of the main substation equipment;

[0110] The image preprocessing unit 62 is used to preprocess the multi-source image data to enhance the defect information;

[0111] The image data fusion unit 63 is used to fuse the preprocessed multi-source image data into a multi-source fusion image through an image fusion algorithm. The multi-source fusion image is obtained by superimposing different images in the multi-source image data based on weights;

[0112] The defect recognition unit 64 is used to analyze the multi-source fusion image through a deep learning model to obtain the defect information;

[0113] The defect visualization unit 65 is used to generate a 3D model of the main substation equipment based on the multi-source image data and the lidar data, and overlay the defect information on the 3D model based on AR.

[0114] As shown Figure 6 The defect detection device 60 for image fusion shown corresponds to Figure 1 The defect detection method shown, and the specific corresponding relationship is as follows:

[0115] The data acquisition unit 61 is used to execute data acquisition based on multiple sensors as shown in step S101.

[0116] The image preprocessing unit 62 is used to execute preprocessing of the acquired multi-source image data as shown in step S102.

[0117] The image data fusion unit 63 is used to execute fusing multi-source image data into multi-source fused images as shown in step S103.

[0118] The defect recognition unit 64 is used to execute analyzing the multi-source fused images to obtain defect information as shown in step S104.

[0119] The defect visualization unit 65 is used to execute generating a three-dimensional model and superimposing the defect information as shown in step S105.

[0120] The specific implementation manners of the above corresponding steps in the embodiments of the present application will not be elaborated herein.

[0121] It can be understood that the functional division of the units schematically shown in the embodiments of the present application is only illustrative and does not constitute a limitation on the functions of the defect detection device 60 for image fusion. In other embodiments of the present application, the defect detection device 60 for image fusion may also adopt different units or a combination of multiple units from those in the above embodiments to implement the functions of the defect detection device 60 for image fusion.

[0122] The defect detection device may be a portable terminal device equipped with or other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), AR devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc.

[0123] Figure 7The figure shows a schematic diagram of the hardware structure of the defect detection device provided by the embodiments of the present application. The defect detection device is used to execute the defect detection method for describing image fusion provided in the foregoing Embodiment 1.

[0124] The defect detection device 700 may include: a processor 710, an external memory interface 720, an internal memory 721, a universal serial bus (USB) interface 730, an antenna 1, an antenna 2, a mobile communication module 750, a wireless communication module 760, a sensor module 780, a button 790, a motor 791, an indicator 792, a camera 793, and a display screen 794, etc. Among them, the sensor module 780 may include a pressure sensor 780A, a gyroscope sensor 780B, a distance sensor 780F, a proximity light sensor 780G, a temperature sensor 780J, a touch sensor 780K, an ambient light sensor 780L, etc.

[0125] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the defect detection device 700. In other embodiments of the present application, the defect detection device 700 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0126] The processor 710 may include one or more processing units. For example, the processor 710 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0127] Among them, the controller may be the nerve center and command center of the defect detection device 700. The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

[0128] A memory may also be provided in the processor 710 for storing instructions and data. In some embodiments, the memory in the processor 710 is a cache memory. This memory may hold instructions or data that the processor 710 has just used or recycled. If the processor 710 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 710, and thus improves the efficiency of the system.

[0129] In some embodiments, the processor 710 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0130] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 710 may include multiple groups of I2C buses. The processor 710 may be respectively coupled to the touch sensor 780K, the charger, the flashlight, the camera 793, etc. through different I2C bus interfaces. For example: the processor 710 may be coupled to the touch sensor 780K through the I2C interface, enabling the processor 710 to communicate with the touch sensor 780K through the I2C bus interface to implement the touch function of the defect detection device 700.

[0131] The I2S interface can be used for audio communication. In some embodiments, the processor 710 may include multiple groups of I2S buses.

[0132] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals.

[0133] The UART interface is a general-purpose serial data bus for asynchronous communication. This bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 710 and the wireless communication module 760. For example, the processor 710 communicates with the Bluetooth module in the wireless communication module 760 through the UART interface to implement the Bluetooth function.

[0134] The MIPI interface can be used to connect the processor 710 with peripheral devices such as the display screen 794 and the camera 793. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 710 and the camera 793 communicate through the CSI interface to implement the shooting function of the defect detection device 700. The processor 710 and the display screen 794 communicate through the DSI interface to implement the display function of the defect detection device 700.

[0135] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 710 with the camera 793, the display screen 794, the wireless communication module 760, the sensor module 780, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0136] The USB interface 730 is an interface that complies with the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 730 can be used to connect a charger to charge the defect detection device 700, and can also be used to transfer data between the defect detection device 700 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other devices, such as AR devices, etc.

[0137] It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present application are only illustrative descriptions and do not constitute a structural limitation on the defect detection device 700. In other embodiments of the present application, the defect detection device 700 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0138] The wireless communication function of the defect detection device 700 can be implemented through antenna 1, antenna 2, the mobile communication module 750, the wireless communication module 760, the modulation and demodulation processor, and the baseband processor, etc.

[0139] Antenna 1 and Antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the defect detection device 700 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, Antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0140] The mobile communication module 750 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the defect detection device 700. The mobile communication module 750 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 750 can receive electromagnetic waves through Antenna 1, perform filtering, amplification, etc. on the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 750 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through Antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 750 can be disposed in the processor 710. In some embodiments, at least some functional modules of the mobile communication module 750 and at least some modules of the processor 710 can be disposed in the same device.

[0141] The wireless communication module 760 can provide solutions for wireless communications including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the defect detection device 700. The wireless communication module 760 can be one or more devices integrating at least one communication processing module. The wireless communication module 760 receives electromagnetic waves through Antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 710. The wireless communication module 760 can also receive the signals to be transmitted from the processor 710, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through Antenna 2 and radiate them out.

[0142] The defect detection device 700 implements the display function through a GPU, a display screen 794, an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 794 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 710 may include one or more GPUs, which execute program instructions to generate or change display information.

[0143] The display screen 794 is used to display images, videos, etc. The display screen 794 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the defect detection device 700 may include 7 or N display screens 794, where N is a positive integer greater than 7.

[0144] The defect detection device 700 can implement the shooting function through an ISP, a camera 793, a video codec, a GPU, a display screen 794, and an application processor, etc.

[0145] The ISP is used to process the data fed back by the camera 793. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera sensor. The optical signal is converted into an electrical signal, and the camera sensor transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise and brightness of the image through algorithms. The ISP can also optimize the exposure parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 793.

[0146] The camera 793 is used to capture static images or videos. An object generates an optical image through a lens and projects it onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard format such as RGB or YUV. In some embodiments, the defect detection device 700 may include N cameras 793, where N is a positive integer greater than 1.

[0147] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the defect detection device 700 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0148] The video codec is used to compress or decompress digital videos. The defect detection device 700 can support one or more video codecs. In this way, the defect detection device 700 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 7, MPEG2, MPEG3, MPEG4, etc.

[0149] The NPU is a neural-network (NN) computing processor. By learning from the biological neural network structure, such as the transmission pattern between human brain neurons, it can quickly process input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the defect detection device 700 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0150] The internal memory 721 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0151] Random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally referred to as DDR5 SDRAM), etc.;

[0152] Non-volatile memory may include disk storage devices and flash memory.

[0153] Flash memory can be classified into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle, and can be classified into single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the number of potential levels of the storage unit. According to the storage specification, it can include universal flash storage (UFS), embedded multi media card (eMMC), etc.

[0154] The random access memory can be directly read and written by the processor 710, and can be used to store the operating system or the executable programs (such as machine instructions) of other running programs, and can also be used to store the data of users and application programs, etc.

[0155] The non-volatile memory can also store executable programs and store the data of users and application programs, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 710.

[0156] The external memory interface 720 can be used to connect to an external non-volatile memory to expand the storage capacity of the defect detection device 700. The external non-volatile memory communicates with the processor 710 through the external memory interface 720 to achieve the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0157] The pressure sensor 780A is used to sense pressure signals and can convert pressure signals into electrical signals. In some embodiments, the pressure sensor 780A may be disposed on the display screen 794. There are many types of pressure sensors 780A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor may include at least two parallel plates having conductive materials. When a force acts on the pressure sensor 780A, the capacitance between the electrodes changes. The defect detection device 700 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 794, the defect detection device 700 detects the intensity of the touch operation according to the pressure sensor 780A. The defect detection device 700 can also calculate the position of the touch according to the detection signal of the pressure sensor 780A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions.

[0158] The gyroscope sensor 780B can be used to determine the motion posture of the defect detection device 700. In some embodiments, the angular velocity of the defect detection device 700 around three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 780B. The gyroscope sensor 780B can be used for anti-shake during shooting.

[0159] The distance sensor 780F is used to measure distance. The defect detection device 700 can measure distance by infrared or laser. In some embodiments, when shooting a scene, the defect detection device 700 can use the distance sensor 780F to measure distance to achieve rapid focusing.

[0160] The proximity light sensor 780G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The defect detection device 700 emits infrared light outward through the light emitting diode. The defect detection device 700 uses the photodiode to detect the infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the defect detection device 700. When insufficient reflected light is detected, the defect detection device 700 can determine that there is no object near the defect detection device 700.

[0161] The ambient light sensor 780L is used to sense the ambient light brightness. The defect detection device 700 can adaptively adjust the brightness of the display screen 794 according to the sensed ambient light brightness. The ambient light sensor 780L can also be used to automatically adjust the white balance during photography.

[0162] The temperature sensor 780J is used to detect temperature. In the embodiments of the present application, the defect detection device 700 uses the temperature sensor 780J to detect the temperature of the surface of the main substation equipment and determine whether there are defects in the main substation equipment.

[0163] The touch sensor 780K, also known as the "touch panel". The touch sensor 780K can be disposed on the display screen 794. The touch sensor 780K and the display screen 794 together form a touch screen, also known as the "touch display screen". The touch sensor 780K is used to detect touch operations acting on it or in its vicinity. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 794. In some other embodiments, the touch sensor 780K can also be disposed on the surface of the defect detection device 700, at a different position from where the display screen 794 is located.

[0164] The keys 790 include a power-on key, volume keys, etc. The keys 790 can be mechanical keys or touch keys. The defect detection device 700 can receive key inputs and generate key signal inputs related to the user settings and function control of the defect detection device 700.

[0165] The motor 791 can generate vibration prompts. The motor 791 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations acting on different regions of the display screen 794 can also correspond to different vibration feedback effects for the motor 791.

[0166] The indicator 792 can be an indicator light and can be used to indicate the charging status, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0167] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In actual applications, the above-mentioned functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above.

[0168] Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0169] In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application.

[0170] Those skilled in the art can understand, Figure 7The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the device to which the solution of this application is applied. The specific device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0171] It should be understood that each step in the above method embodiments provided by this application can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The method steps disclosed in combination with the embodiments of this application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor.

[0172] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps in any one of the foregoing embodiments are implemented.

[0173] This application also provides a computer program product, which includes: a computer program (which can also be called code, or instruction). When the computer program is run, the computer is made to execute the method in any one of the above embodiments.

[0174] Embodiment 3

[0175] As Figure 8 shown, this application also provides a computer-readable storage medium, which stores a computer program (which can also be called code, or instruction). When the computer program is run, the computer is made to execute the method in any one of the foregoing embodiments.

[0176] The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk (SSD)).

[0177] This application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the device in any one of the above embodiments.

[0178] In a possible design, the chip system further includes a memory for storing program instructions and data, and the memory is located inside or outside the processor.

[0179] The chip system can be composed of chips or can include chips and other discrete devices.

[0180] In some embodiments, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor that implements functions by reading software code stored in a memory.

[0181] In some embodiments, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately provided from the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transitory processor, such as a read-only memory (ROM). It may be integrated with the processor on the same chip or may be separately provided on different chips. The embodiments of the present application do not specifically limit the type of the memory and the setting manner of the memory and the processor.

[0182] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a DSP, a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0183] The various embodiments of the present application can be combined arbitrarily to achieve different technical effects. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0184] In the foregoing embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0185] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be completed by relevant hardware instructed by a computer program. This program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the foregoing method embodiments. The foregoing storage medium includes various media that can store program codes, such as read-only memory ROM or random access memory RAM, magnetic disks, or optical discs.

[0186] In summary, the above description is only an embodiment of the technical solution of this application and is not intended to limit the protection scope of this application. Any modifications, equivalent replacements, improvements, etc. made according to the disclosure of this application shall be included within the protection scope of this application.

Claims

1. A defect detection method for image fusion, characterized in that, A defect detection device applied to detect defects of main substation equipment, the method includes: Collect multi-source image data and lidar data of the main substation equipment through a variety of sensors, and the defect information of the main substation equipment is included in the multi-source image data; Preprocess the multi-source image data to enhance the defect information; Fuse the preprocessed multi-source image data into a multi-source fusion image through an image fusion algorithm, and the multi-source fusion image is obtained by superimposing different images in the multi-source image data based on weights; Analyze the multi-source fusion image through a deep learning model to obtain the defect information; Generate a three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimpose the defect information on the three-dimensional model based on augmented reality (AR).

2. The method according to claim 1, characterized in that, The variety of sensors include a high-definition camera, an infrared thermal imager, a depth camera, and a lidar, and the multi-source image data includes visible light images, infrared thermal imaging images, and depth images.

3. The method according to claim 1, wherein The preprocessing of the multi-source image data to enhance the defect information includes: Perform denoising processing, image alignment, color correction, and image enhancement on the multi-source image data to enhance the defect information; Among them, the denoising processing is used to remove the noise in the multi-source image data and improve the clarity of the multi-source image data; the image alignment is used to place the multi-source image data under the same coordinates; the color correction is used to ensure that the colors of the multi-source image data from different sources are consistent; the image enhancement is used to highlight the details on the surface of the main substation equipment in the multi-source image data.

4. The method according to claim 1, wherein The image fusion algorithm includes an algorithm for determining the weights of images from different sources based on image quality, sensor characteristics, and defect characteristics, and fusing the images from different sources according to the weights to generate a fusion image.

5. The method according to claim 1, wherein The deep learning model includes a convolutional neural network (CNN). The deep learning model is used to determine the defect type, defect location, and defect severity of the main substation equipment. The defect types include cracks, corrosion, overheating, and surface damage.

6. The method according to claim 5, wherein The deep learning model includes an image defect quality assessment algorithm, and the image defect quality assessment algorithm includes the following steps: Step (1). Extract multiple image features from the detected image defect area. The image features include the area of the image defect area, the sharpness of the edge of the image defect area, the regularity of the shape of the image defect area, and the surface texture of the image defect area; Step (2). Based on the extracted image features, calculate a severity score in combination with the geometric attributes and surface features of the image defect area. The severity score is used to divide the severity level of the image defect area; Step (3). By comparing with the standard equipment model, exclude false detections caused by factors such as light, shadow, and noise, and determine the image defect area of the real damage; Step (4). Based on the severity score, sort the image defect areas of the real damage in descending order, and assign repair priorities from high to low in turn. The repair priority is used to indicate the processing order of the image defect areas.

7. The method according to claim 1, wherein The three-dimensional model includes geometric shapes and surface features indicating the main substation equipment, and the defect information includes specific spatial positions of defects on the three-dimensional model. After generating the three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimposing the defect information on the three-dimensional model based on augmented reality (AR), the method further includes: Showing, through the AR, repair solutions and steps for defects on the main substation equipment.

8. An image fusion-based defect detection device, characterized in that, Including: A data acquisition unit configured to acquire multi-source image data and lidar data of the main substation equipment through multiple sensors, wherein the multi-source image data contains defect information of the main substation equipment; An image preprocessing unit configured to preprocess the multi-source image data to enhance the defect information; An image data fusion unit configured to fuse the preprocessed multi-source image data into a multi-source fused image through an image fusion algorithm, where the multi-source fused image is obtained by weighted superposition of different images in the multi-source image data; A defect recognition unit configured to analyze the multi-source fused image through a deep learning model to obtain the defect information; A defect visualization unit configured to generate the three-dimensional model of the main substation equipment based on the multi-source image data and the lidar data, and superimpose the defect information on the three-dimensional model based on AR.

9. A computer device, including computer program instructions, which, when running on the computer device, cause the computer device to execute the defect detection method for image fusion according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that, When the computer instructions are executed by a processor, the defect detection method for image fusion according to any one of claims 1 to 7 is implemented.

Images