Image data processing method and device, storage medium and electronic device

Abstract

The invention discloses an image data processing method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring a to-be-processed image obtained by performing image acquisition on a photovoltaic power station; extracting the thermal infrared image by using an object extraction model to obtain a first photovoltaic module image in the thermal infrared image, and extracting the visible light image by using the object extraction model to obtain a second photovoltaic module image in the visible light image; image registration is carried out on the first photovoltaic module image and the second photovoltaic module image, a registered photovoltaic module image pair is obtained, and in the photovoltaic module image pair, the angular point coordinates of the registered first photovoltaic module image and the angular point coordinates of the registered second photovoltaic module image have a corresponding relation; and performing fault detection on a target photovoltaic module indicated by the photovoltaic module image pair, and sending a detection result obtained by fault detection to the target object.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to an image data processing method and device, a storage medium and an electronic device. Background Art

[0002] With the rapid development of renewable energy technology, photovoltaic energy has attracted widespread attention due to its clean and renewable characteristics. In particular, photovoltaic systems that use photovoltaic energy to generate electricity have great application prospects. When generating photovoltaic power, image registration is often required to improve the monitoring accuracy of the photovoltaic system. Existing image registration usually relies on the grayscale features or texture features of the image for registration. When the lighting conditions change or there is dirt on the surface of the photovoltaic panel, the performance will be significantly reduced.

[0003] In the related art, it is proposed to introduce machine learning algorithms to improve the accuracy and robustness of registration. However, these methods often require a large amount of training data, and in practical applications, performance may be unstable due to environmental changes. At the same time, the existing image registration devices also have shortcomings in hardware design, such as the selection of sensors, the layout of image acquisition modules, etc., which may affect the final registration effect. Although there are also methods for image registration using edge detection methods such as the Canny edge detection algorithm, the Canny algorithm may cause false detection or missed detection when processing images with large noise or discontinuous edges. In addition, there are also methods for image registration using algorithms such as SIFT (Scale Invariant Feature Transform), but the SIFT algorithm is complex to calculate and cannot be paired with the features of bi-optical images. It can be seen that the current image registration accuracy is low.

[0004] Regarding the related technologies, there is a technical problem of how to improve the accuracy of image registration, and no effective solution has been proposed yet.

[0005] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0006] The embodiments of the present application provide a method and device for processing image data, a storage medium, and an electronic device to at least solve the technical problem of how to improve the accuracy of image registration.

[0007] According to one aspect of an embodiment of the present application, a method for processing image data is provided, comprising: acquiring an image to be processed obtained by performing image acquisition on a photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image; extracting the thermal infrared image using an object extraction model to obtain a first photovoltaic component image in the thermal infrared image, and extracting the visible light image using an object extraction model to obtain a second photovoltaic component image in the visible light image; performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, wherein in the photovoltaic component image pair, there is a corresponding relationship between the corner point coordinates of the first photovoltaic component image after registration and the corner point coordinates of the second photovoltaic component image after registration; performing fault detection on a target photovoltaic component indicated by the photovoltaic component image pair, and sending the detection result obtained by the fault detection to the target object.

[0008] In an exemplary embodiment, obtaining an image to be processed obtained by acquiring images of a photovoltaic power station includes: controlling a drone to fly according to an inspection route, and controlling a dual-light camera configured on the drone to acquire images of an image acquisition area of ​​the photovoltaic power station to obtain a dual-light image, wherein the image acquisition area includes different photovoltaic components, and the dual-light image includes initial images of the different photovoltaic components, and the initial image of each photovoltaic component has a different image type; determining a component image belonging to the target photovoltaic component from the initial image, and determining the image to be processed according to the component image.

[0009] In an exemplary embodiment, before performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, it includes: performing edge extraction on the first photovoltaic component image to obtain a plurality of first edge lines; performing rectangular fitting on the plurality of first edge lines to obtain a first fitting image; filtering the first fitting image according to an image filtering condition to obtain a first filtered image; and updating the first filtered image to the first photovoltaic component image.

[0010] In an exemplary embodiment, performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair includes: performing coordinate extraction on the first photovoltaic component image to obtain a first group of corner point coordinates of the first photovoltaic component image, wherein the first group of corner point coordinates includes at least one of the following: a first upper left corner point coordinate, a first lower left corner point coordinate, a first upper right corner point coordinate, and a first lower right corner point coordinate; and performing coordinate extraction on the second photovoltaic component image to obtain a second group of corner point coordinates of the second photovoltaic component image, wherein the second group of corner point coordinates includes at least one of the following: a second upper left corner point coordinate, a second lower left corner point coordinate, a second upper right corner point coordinate, and a second lower right corner point coordinate; performing corner point matching on the first group of corner point coordinates and the second group of corner point coordinates using an iterative nearest point algorithm to obtain a coordinate correspondence relationship between the first group of corner point coordinates and the second group of corner point coordinates; mapping the first group of corner point coordinates to the coordinate axis where the second group of corner point coordinates are located using the coordinate correspondence relationship to obtain a third group of corner point coordinates; and determining the photovoltaic component image pair based on the second group of corner point coordinates and the third group of corner point coordinates.

[0011] In an exemplary embodiment, after performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, the method further includes: performing image fusion on the registered first photovoltaic component image to obtain a first fused image, performing an image enhancement operation on the first fused image to obtain a first target image; and performing image fusion on the registered second photovoltaic component image to obtain a second fused image, performing an image enhancement operation on the second fused image to obtain a second target image; and in the case of determining that the first photovoltaic component indicated by the first target image and the second photovoltaic component indicated by the second target image are the same photovoltaic component, determining the same photovoltaic component as the target photovoltaic component.

[0012] In an exemplary embodiment, fault detection is performed on a target photovoltaic component indicated by the photovoltaic component image pair, and the detection result obtained by the fault detection is sent to a target object, including: inputting the photovoltaic component image pair into a fault detection model to obtain an image fault type output by the fault detection model, wherein the fault detection model is trained using historical photovoltaic component image pairs as input samples and image fault types corresponding to images of the historical photovoltaic component image pairs as output samples; determining the detection result based on the image fault type, and sending the detection result to the target object.

[0013] According to another aspect of an embodiment of the present application, there is also provided an image data processing device, including: an acquisition module, used to acquire an image to be processed obtained by performing image acquisition on a photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image; an extraction module, used to extract the thermal infrared image using an object extraction model to obtain a first photovoltaic component image in the thermal infrared image, and to extract the visible light image using an object extraction model to obtain a second photovoltaic component image in the visible light image; a registration module, used to perform image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, wherein in the photovoltaic component image pair, there is a corresponding relationship between the corner point coordinates of the first photovoltaic component image after registration and the corner point coordinates of the second photovoltaic component image after registration; and a sending module, used to perform fault detection on a target photovoltaic component indicated by the photovoltaic component image pair, and send the detection result obtained by the fault detection to the target object.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for processing image data when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image data processing method through the computer program.

[0016] According to another aspect of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0017] According to the present application, by acquiring an image to be processed obtained by image acquisition of a photovoltaic power station, an object extraction model is used to extract a thermal infrared image of the image to be processed to obtain a first photovoltaic component image in the thermal infrared image, and an object extraction model is used to extract a visible light image of the image to be processed to obtain a second photovoltaic component image in the visible light image; the first photovoltaic component image and the second photovoltaic component image are image registered to obtain a photovoltaic component image pair after registration, wherein in the photovoltaic component image pair, there is a corresponding relationship between the corner point coordinates of the first photovoltaic component image after registration and the corner point coordinates of the second photovoltaic component image after registration; fault detection is performed on the target photovoltaic component indicated by the photovoltaic component image pair, and the detection result obtained by performing the fault detection is sent to the target object. It can be seen that the present application can perform image registration based on edge rectangle features. In short, by detecting the rectangular edges in the dual light image and using the inherent shape features of the photovoltaic component, fast and accurate registration of images of different wavelengths is achieved, which not only solves the technical problem of how to improve the accuracy of image registration, but also improves the efficiency and accuracy of image registration, and also realizes real-time monitoring of photovoltaic components, thereby improving the efficiency of fault detection. Moreover, it can work stably under various environmental conditions, providing an effective technical means for the monitoring and management of photovoltaic systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a hardware structure block diagram of a mobile terminal of an image data processing method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for processing image data according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of the principle of image registration according to an embodiment of the present application;

[0023] Figure 4 It is a structural block diagram of an image data processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image data processing method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Central Processing Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for processing image data in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for processing image data is provided. Figure 2 is a flowchart of a method for processing image data according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps:

[0030] Step S202: acquiring an image to be processed obtained by collecting images of the photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image;

[0031] Step S204: extracting the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and extracting the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image;

[0032] Optionally, the object extraction model is trained by using unlabeled images containing photovoltaic components as input samples and labeled images containing photovoltaic components as output samples. The unlabeled image represents an original image captured of the photovoltaic component, and the labeled image represents an image in which the photovoltaic component is marked on the original image by a marking box or the like.

[0033] Step S206: performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration;

[0034] Step S208: performing fault detection on the target photovoltaic component indicated by the photovoltaic component image, and sending the detection result obtained by the fault detection to the target object.

[0035] Through the above steps, by acquiring the image to be processed obtained by image acquisition of the photovoltaic power station, the thermal infrared image of the image to be processed is extracted using the object extraction model to obtain the first photovoltaic component image in the thermal infrared image, and the visible light image of the image to be processed is extracted using the object extraction model to obtain the second photovoltaic component image in the visible light image; the first photovoltaic component image and the second photovoltaic component image are image registered to obtain a photovoltaic component image pair after registration, wherein, in the photovoltaic component image pair, there is a corresponding relationship between the corner point coordinates of the first photovoltaic component image after registration and the corner point coordinates of the second photovoltaic component image after registration; fault detection is performed on the target photovoltaic component indicated by the photovoltaic component image pair, and the detection result obtained by performing the fault detection is sent to the target object. It can be seen that the present application can perform image registration based on edge rectangular features. In short, by detecting the rectangular edges in the dual-light image and using the inherent shape features of the photovoltaic component, fast and accurate registration of images of different wavelengths is achieved, which not only solves the technical problem of how to improve the accuracy of image registration, but also improves the efficiency and accuracy of image registration, and also realizes real-time monitoring of photovoltaic components, thereby improving the efficiency of fault detection. Moreover, it can work stably under various environmental conditions, providing an effective technical means for the monitoring and management of photovoltaic systems.

[0036] In an exemplary embodiment, the implementation process of obtaining the image to be processed obtained by image acquisition of the photovoltaic power station in step S202 specifically includes: controlling the drone to fly according to the inspection route, and at the same time controlling the dual-light camera configured on the drone to acquire images of the image acquisition area of ​​the photovoltaic power station to obtain a dual-light image, wherein the image acquisition area includes different photovoltaic components, and the dual-light image includes initial images of the different photovoltaic components, and the initial image of each photovoltaic component has a different image type; determining the component image belonging to the target photovoltaic component from the initial image, and determining the image to be processed according to the component image.

[0037] The fact that the initial image of each photovoltaic module has different image types can be understood to include at least a thermal infrared image type and a visible light image type supported by the dual-light camera.

[0038] In an exemplary embodiment, before performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, the method further includes: performing edge extraction on the first photovoltaic component image to obtain a plurality of first edge lines; performing rectangular fitting on the plurality of first edge lines to obtain a first fitting image; filtering the first fitting image according to an image filtering condition to obtain a first filtered image; and updating the first filtered image to the first photovoltaic component image. Through the filtering operation of this embodiment, incomplete photovoltaic component images and photovoltaic component images with segmentation errors can be effectively removed.

[0039] The above-mentioned image screening conditions may be, for example, the length, width, and ratio index between the length and width of the photovoltaic component image.

[0040] In an exemplary embodiment, the technical solution for performing image registration on the first photovoltaic component image and the second photovoltaic component image in step S206 to obtain a pair of registered photovoltaic component images includes the following steps: extracting coordinates of the first photovoltaic component image to obtain a first set of corner point coordinates of the first photovoltaic component image, wherein the first set of corner point coordinates includes at least one of the following: a first upper left corner point coordinate, a first lower left corner point coordinate, a first upper right corner point coordinate, and a first lower right corner point coordinate; and extracting coordinates of the second photovoltaic component image to obtain a second set of corner point coordinates of the second photovoltaic component image. Corner point coordinates, wherein the second group of corner point coordinates includes at least one of the following: a second upper left corner point coordinate, a second lower left corner point coordinate, a second upper right corner point coordinate, and a second lower right corner point coordinate; using an iterative closest point algorithm to perform corner point matching on the first group of corner point coordinates and the second group of corner point coordinates to obtain a coordinate correspondence relationship between the first group of corner point coordinates and the second group of corner point coordinates; using the coordinate correspondence relationship to map the first group of corner point coordinates to the coordinate axis where the second group of corner point coordinates are located to obtain a third group of corner point coordinates; determining the photovoltaic component image pair based on the second group of corner point coordinates and the third group of corner point coordinates.

[0041] Optionally, the coordinate correspondence relationship can be obtained by calculating the image rotation matrix and the image translation matrix corresponding to the first photovoltaic component image and the second photovoltaic component image, respectively. In this way, by performing a matrix transformation on the thermal infrared image of the first photovoltaic component image or the second photovoltaic component image, the first photovoltaic component image and the second photovoltaic component image can be spatially aligned using the coordinate correspondence relationship and displayed on the same coordinate axis.

[0042] Among them, the Iterative Closest Point (ICP) algorithm is a key algorithm widely used in computer vision tasks, especially in 3D surface registration and Simultaneous Localization And Mapping (SLAM) tasks. It can align 3D surfaces based on geometry without relying on depth or structural information. The core idea of ​​the ICP algorithm is to gradually minimize the distance between point clouds through an iterative process, so as to achieve accurate alignment of point clouds. Specifically, for two sets of point cloud data, by finding the closest point of each point in the other set of point clouds, and calculating a transformation (including rotation and translation), the source point cloud is brought closer to the target point cloud. This process is iterated until certain convergence conditions are met.

[0043] Considering that the ICP algorithm itself has some limitations in practical applications, such as being prone to falling into local optimal solutions, being sensitive to initial values, and having robustness issues in the face of noise and outliers. The present application can further utilize variant algorithms of the ICP algorithm for image registration, wherein the variant algorithms include, for example, point-to-line ICP (point-to-line ICP) algorithms, point-to-plane ICP (point-to-plane ICP) algorithms, and algorithms combined with the Normal Distributions Transform (NDT) algorithm. In addition, the ICP algorithm has applications in multimodal image registration, robot vision, medical image processing, and other fields. For example, in medical image processing, the ICP algorithm can be used to optimize the selection and registration of feature points. In automated building detection, the ICP algorithm is also used for image recognition and registration of three-dimensional models.

[0044] Similarly, before using the iterative closest point algorithm to perform image registration on the first photovoltaic component image and the second photovoltaic component image to obtain the registered photovoltaic component image pair, edge extraction can also be performed on the second photovoltaic component image to obtain multiple second edge lines; rectangular fitting is performed on the multiple second edge lines to obtain a second fitting image; the second fitting image is filtered according to the image filtering condition to obtain a second filtered image; and the second filtered image is updated to the second photovoltaic component image.

[0045] Based on the above embodiments, the present application utilizes a convolutional neural network for image recognition and feature extraction, and combines it with an ICP algorithm for image registration, thereby improving the registration effect of dual-light images of photovoltaic modules, and thereby improving the fault detection accuracy of photovoltaic modules.

[0046] In an exemplary embodiment, after performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a registered photovoltaic component image pair, further, image fusion can be performed on the registered first photovoltaic component image to obtain a first fused image, and an image enhancement operation can be performed on the first fused image to obtain a first target image; and image fusion can be performed on the registered second photovoltaic component image to obtain a second fused image, and an image enhancement operation can be performed on the second fused image to obtain a second target image; in the case where it is determined that the first photovoltaic component indicated by the first target image and the second photovoltaic component indicated by the second target image are the same photovoltaic component, the same photovoltaic component is determined as the target photovoltaic component.

[0047] In an exemplary embodiment, the process of performing fault detection on the target photovoltaic component indicated by the photovoltaic component image pair and sending the detection result obtained by the fault detection to the target object is implemented by the following steps: inputting the photovoltaic component image pair into a fault detection model to obtain the image fault type output by the fault detection model, wherein the fault detection model is trained with historical photovoltaic component image pairs as input samples and image fault types corresponding to the images of the historical photovoltaic component image pairs as output samples; determining the detection result based on the image fault type and sending it to the target object. This embodiment can monitor photovoltaic components in real time and provide real-time feedback of the fault detection results of photovoltaic components to users, thereby improving the robustness of photovoltaic components.

[0048] Obviously, the embodiments described above are only some embodiments of the present application, not all embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present application, specifically:

[0049] In an optional embodiment, the present application is based on Figure 3 The registration principle shown in FIG. 1 illustrates the registration process of the bifocal image. The specific steps of this process are as follows:

[0050] The first step is to use dual-light cameras (thermal infrared cameras and visible light cameras) to inspect photovoltaic power plants. Specifically, a drone equipped with a dual-light camera is used to inspect and take photos of photovoltaic modules in the photovoltaic power plant to obtain thermal infrared images and visible light images.

[0051] In the second step, a convolutional neural network is used to automatically identify photovoltaic modules in thermal infrared images and visible light images to obtain photovoltaic module images in thermal infrared images and visible light images.

[0052] The third step is to segment the location of the photovoltaic components in the dual-light image. For example, a convolutional neural network can be used to segment the photovoltaic components in the image. Specifically, the edge of the photovoltaic component image is extracted and rectangular fitting is performed. Based on the fitting, the components are screened and the incomplete photovoltaic component images and the wrongly segmented photovoltaic component images are removed by screening based on the length, width and the ratio between them.

[0053] The fourth step is to use rectangle fitting to extract the corner coordinates of the photovoltaic module. Specifically, based on the rectangular edge of the photovoltaic module image, the coordinates of the four corner points of the rectangle are extracted to provide key feature points for subsequent image matching.

[0054] The fifth step is to use the ICP algorithm to match the corner point coordinates. The rectangular corner points in the thermal infrared image and the visible light image are matched. The ICP algorithm can find corresponding points in different images, thereby achieving accurate correspondence of the corner points.

[0055] The sixth step is to calculate the rotation and translation relationship between the two images through mathematical methods based on the matched corner points, that is, to calculate the image rotation and translation matrices corresponding to the thermal infrared image and the visible light image respectively, to provide the necessary transformation parameters for image registration.

[0056] The seventh step is to use the matched corner coordinates to achieve image matching. According to the calculated rotation and translation matrices, the thermal infrared image or visible light image is transformed to achieve spatial alignment of the two images, thereby achieving the purpose of image registration.

[0057] The eighth step is to post-process the registered images, including image fusion, texture enhancement and other operations, to improve the visual effect of the image and the accuracy of subsequent analysis.

[0058] It can be seen that compared with traditional image registration technology, this application uses CNN for image recognition and feature extraction, which improves the accuracy of photovoltaic component recognition and the robustness of edge extraction. At the same time, the ICP algorithm is used for corner matching to improve the accuracy and efficiency of image registration, and realizes fast and accurate image processing, providing strong technical support for the monitoring and management of photovoltaic power stations.

[0059] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0060] In this embodiment, a device for processing image data is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0061] Figure 4 is a structural block diagram of an image data processing device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0062] An acquisition module 42 is used to acquire an image to be processed obtained by collecting images of the photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image;

[0063] An extraction module 44 is used to extract the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and to extract the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image;

[0064] A registration module 46, configured to perform image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration;

[0065] The sending module 48 is used to perform fault detection on the target photovoltaic component indicated by the photovoltaic component image pair, and send the detection result obtained by the fault detection to the target object.

[0066] Through the above-mentioned device, by acquiring the image to be processed obtained by image acquisition of the photovoltaic power station, the thermal infrared image of the image to be processed is extracted using the object extraction model to obtain the first photovoltaic component image in the thermal infrared image, and the visible light image of the image to be processed is extracted using the object extraction model to obtain the second photovoltaic component image in the visible light image; the first photovoltaic component image and the second photovoltaic component image are image-aligned to obtain a photovoltaic component image pair after alignment, wherein, in the photovoltaic component image pair, there is a corresponding relationship between the corner point coordinates of the first photovoltaic component image after alignment and the corner point coordinates of the second photovoltaic component image after alignment; fault detection is performed on the target photovoltaic component indicated by the photovoltaic component image pair, and the detection result obtained by the fault detection is sent to the target object. It can be seen that the present application can perform image alignment based on edge rectangular features. In short, by detecting the rectangular edges in the dual-light image and using the inherent shape features of the photovoltaic component, fast and accurate alignment of images of different wavelengths is achieved, which not only solves the technical problem of how to improve the accuracy of image alignment, but also improves the efficiency and accuracy of image alignment, and also realizes real-time monitoring of photovoltaic components, thereby improving the efficiency of fault detection. Moreover, it can work stably under various environmental conditions, providing an effective technical means for the monitoring and management of photovoltaic systems.

[0067] In an exemplary embodiment, the acquisition module 42 is also used to: control the UAV to fly according to the inspection route, and at the same time control the dual-light camera configured on the UAV to collect images of the image acquisition area of ​​the photovoltaic power station to obtain a dual-light image, wherein the image acquisition area includes different photovoltaic components, and the dual-light image includes initial images of the different photovoltaic components, and the initial image of each photovoltaic component has a different image type; determine the component image belonging to the target photovoltaic component from the initial image, and determine the image to be processed according to the component image.

[0068] In an exemplary embodiment, the extraction module 44 is further used to perform edge extraction on the first photovoltaic component image to obtain a plurality of first edge lines before performing image registration on the first photovoltaic component image and the second photovoltaic component image to obtain a pair of registered photovoltaic component images; perform rectangular fitting on the plurality of first edge lines to obtain a first fitting image; filter the first fitting image according to an image filtering condition to obtain a first filtered image; and update the first filtered image to the first photovoltaic component image.

[0069] In an exemplary embodiment, the registration module 46 is further used to: extract coordinates of the first photovoltaic component image to obtain a first set of corner point coordinates of the first photovoltaic component image, wherein the first set of corner point coordinates includes at least one of the following: a first upper left corner point coordinate, a first lower left corner point coordinate, a first upper right corner point coordinate, and a first lower right corner point coordinate; and extract coordinates of the second photovoltaic component image to obtain a second set of corner point coordinates of the second photovoltaic component image, wherein the second set of corner point coordinates includes at least one of the following: a second upper left corner point coordinate, a second lower left corner point coordinate, a second upper right corner point coordinate, and a second lower right corner point coordinate; perform corner point matching on the first set of corner point coordinates and the second set of corner point coordinates using an iterative nearest point algorithm to obtain a coordinate correspondence relationship between the first set of corner point coordinates and the second set of corner point coordinates; map the first set of corner point coordinates to the coordinate axis where the second set of corner point coordinates are located using the coordinate correspondence relationship to obtain a third set of corner point coordinates; and determine the photovoltaic component image pair based on the second set of corner point coordinates and the third set of corner point coordinates.

[0070] In an exemplary embodiment, the registration module 46 is further used to perform image fusion on the first photovoltaic component image after registration with the second photovoltaic component image to obtain a registered photovoltaic component image pair, obtain a first fused image, perform image enhancement on the first fused image, and obtain a first target image; and perform image fusion on the second photovoltaic component image after registration to obtain a second fused image, perform image enhancement on the second fused image, and obtain a second target image; when it is determined that the first photovoltaic component indicated by the first target image and the second photovoltaic component indicated by the second target image are the same photovoltaic component, determine the same photovoltaic component as the target photovoltaic component.

[0071] In an exemplary embodiment, the sending module 48 is also used to: input the photovoltaic component image pair into a fault detection model to obtain an image fault type output by the fault detection model, wherein the fault detection model is trained using historical photovoltaic component image pairs as input samples and image fault types corresponding to the images of the historical photovoltaic component image pairs as output samples; determine the detection result based on the image fault type, and send it to the target object.

[0072] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0073] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0074] S1, acquiring an image to be processed obtained by collecting images of a photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image;

[0075] S2, extracting the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and extracting the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image;

[0076] S3, performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration;

[0077] S4, performing fault detection on the target photovoltaic component indicated by the photovoltaic component image, and sending the detection result obtained by the fault detection to the target object.

[0078] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0079] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0080] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0081] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0082] S1, acquiring an image to be processed obtained by collecting images of a photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image;

[0083] S2, extracting the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and extracting the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image;

[0084] S3, performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration;

[0085] S4, performing fault detection on the target photovoltaic component indicated by the photovoltaic component image, and sending the detection result obtained by the fault detection to the target object.

[0086] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0087] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0088] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0089] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0090] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0091] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0092] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing image data, characterized in that: include: Acquire an image to be processed obtained by collecting images of the photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image; Extracting the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and extracting the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image; Performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration; Fault detection is performed on the target photovoltaic component indicated by the photovoltaic component image, and a detection result obtained by the fault detection is sent to the target object.

2. The method according to claim 1, characterized in that The image to be processed obtained by collecting images of the photovoltaic power station includes: Controlling the drone to fly according to the inspection route, and controlling the dual-light camera configured on the drone to collect images of the image collection area of ​​the photovoltaic power station to obtain a dual-light image, wherein the image collection area includes different photovoltaic components, and the dual-light image includes initial images of the different photovoltaic components, and each initial image of the photovoltaic component has a different image type; A component image belonging to the target photovoltaic component is determined from the initial image, and the image to be processed is determined according to the component image.

3. The method according to claim 1, characterized in that Before performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, the method includes: Performing edge extraction on the first photovoltaic assembly image to obtain a plurality of first edge lines; Performing rectangular fitting on the plurality of first edge lines to obtain a first fitting image; Filtering the first fitting image according to an image filtering condition to obtain a first filtering image; The first screening image is updated to the first photovoltaic assembly image.

4. The method according to claim 1, characterized in that: Performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair includes: Extracting coordinates of the first photovoltaic assembly image to obtain a first set of corner point coordinates of the first photovoltaic assembly image, wherein the first set of corner point coordinates includes at least one of the following: a first upper left corner point coordinate, a first lower left corner point coordinate, a first upper right corner point coordinate, and a first lower right corner point coordinate; and extracting coordinates of the second photovoltaic assembly image to obtain a second set of corner point coordinates of the second photovoltaic assembly image, wherein the second set of corner point coordinates includes at least one of the following: a second upper left corner point coordinate, a second lower left corner point coordinate, a second upper right corner point coordinate, and a second lower right corner point coordinate; Performing corner point matching on the first set of corner point coordinates and the second set of corner point coordinates using an iterative closest point algorithm to obtain a coordinate correspondence relationship between the first set of corner point coordinates and the second set of corner point coordinates; Mapping the first set of corner point coordinates to the coordinate axis where the second set of corner point coordinates are located using the coordinate correspondence relationship to obtain a third set of corner point coordinates; The photovoltaic assembly image pair is determined based on the second set of corner point coordinates and the third set of corner point coordinates.

5. The method according to claim 4, characterized in that After performing image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, the method further includes: Performing image fusion on the registered first photovoltaic assembly image to obtain a first fused image, and performing an image enhancement operation on the first fused image to obtain a first target image; and performing image fusion on the registered second photovoltaic assembly image to obtain a second fused image, and performing an image enhancement operation on the second fused image to obtain a second target image; When it is determined that the first photovoltaic component indicated by the first target image and the second photovoltaic component indicated by the second target image are the same photovoltaic component, the same photovoltaic component is determined as the target photovoltaic component.

6. The method according to claim 1, characterized in that Performing fault detection on a target photovoltaic component indicated by the photovoltaic component image, and sending a detection result obtained by performing the fault detection to a target object, includes: Inputting the photovoltaic module image pair into a fault detection model to obtain an image fault type output by the fault detection model, wherein the fault detection model is trained using historical photovoltaic module image pairs as input samples and image fault types corresponding to images of the historical photovoltaic module image pairs as output samples; The detection result is determined based on the image fault type and sent to a target object.

7. An image data processing device, characterized in that: include: An acquisition module, used to acquire an image to be processed obtained by collecting images of the photovoltaic power station, wherein the image to be processed includes at least a thermal infrared image and a visible light image; an extraction module, configured to extract the thermal infrared image using an object extraction model to obtain a first photovoltaic assembly image in the thermal infrared image, and to extract the visible light image using an object extraction model to obtain a second photovoltaic assembly image in the visible light image; a registration module, configured to perform image registration on the first photovoltaic assembly image and the second photovoltaic assembly image to obtain a registered photovoltaic assembly image pair, wherein in the photovoltaic assembly image pair, there is a corresponding relationship between the coordinates of the corner points of the first photovoltaic assembly image after registration and the coordinates of the corner points of the second photovoltaic assembly image after registration; The sending module is used to perform fault detection on the target photovoltaic component indicated by the photovoltaic component image pair, and send the detection result obtained by the fault detection to the target object.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images