Photovoltaic inspection image registration method, device and system

Abstract

The invention discloses a photovoltaic inspection image registration method, device and system, and belongs to the field of image processing. The method comprises the following steps: firstly, obtaining an initial affine matrix according to the distance from a shooting point to a shooting target, then cutting a preliminary cut small image with the same content as an infrared image based on the initial affine matrix, and then respectively extracting string information on the infrared image and the preliminary cut small image; and then infrared string information is mapped to the preliminarily cut small image to obtain mapping string information. And obtaining an optimal deviation value based on the deviation of the mapping string information and the visible light string information, then adjusting the initial affine matrix based on the optimal deviation value to obtain a target affine matrix, and finally performing image registration according to the target affine matrix. According to the scheme, after the initial affine matrix is obtained, the deviation of the initial affine matrix is determined according to the string information on the image, then the initial affine matrix is adjusted based on the deviation, and the finally obtained target affine matrix is more accurate during image registration.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a photovoltaic inspection image registration method, device and system. Background Art

[0002] In photovoltaic inspections, dual-light images are generally taken for defect detection. The two images can realize different types of defect detection. How to achieve the defect fusion of the two is very important in photovoltaic inspections.

[0003] The existing solution is to use the drone to hover and take pictures to achieve the mapping between the two. Because the center points of the two images coincide when hovering, the affine transformation can be used to achieve both mapping and fusion. However, hovering and taking pictures will affect the inspection efficiency because each waypoint needs to hover, which will increase the inspection time, increase the loss of the drone battery, and increase the inspection cost.

[0004] In order to reduce the inspection time, reduce the loss of drone batteries, and improve the inspection efficiency, a method for drones to take photos without hovering is proposed. However, since there is a time interval between the infrared image and the visible light image when taking photos without hovering, it is necessary to align the infrared image and the visible light image. The existing technology uses the positioning principle to obtain the affine matrix and performs image registration based on the affine matrix. However, the affine matrix obtained based on the positioning principle is inaccurate, and the final image registration error is large, which cannot be applied in practice. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present application provides a photovoltaic inspection image registration method, device and system to solve the problem that the prior art uses a drone to take pictures without hovering, uses the positioning principle to obtain an affine matrix, and performs image registration based on the affine matrix, but the affine matrix obtained based on the positioning principle is inaccurate, the final image registration error is large, and it cannot be used in practice.

[0006] The technical solution adopted by this application to solve its technical problem is:

[0007] In a first aspect, a photovoltaic inspection image registration method is provided, comprising:

[0008] Acquire a first distance and a second distance, wherein the first distance is the distance from the infrared image shooting point to the shooting target, and the second distance is the distance from the visible light image shooting point to the same shooting target, and the content on the visible light image includes the content on the infrared image;

[0009] Obtaining an initial affine matrix based on the first distance and the second distance;

[0010] Cropping the visible light image based on the initial affine matrix to obtain a preliminary cropped thumbnail with the same content as the infrared image;

[0011] Extracting infrared group string information and visible light group string information from the infrared image and the preliminary cropped thumbnail respectively;

[0012] Mapping the infrared group string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped group string information;

[0013] Obtaining an optimal deviation value based on the mapping string information and the visible light string information;

[0014] Adjusting the initial affine matrix based on the optimal deviation value to obtain a target affine matrix;

[0015] The infrared image and the visible light image are registered based on the target affine matrix.

[0016] Further, obtaining the optimal deviation value based on the mapping string information and the visible light string information includes:

[0017] Sorting the mapping group string information according to the distance from the mapping group string information to the center point of the preliminary cropped thumbnail;

[0018] The frame of the mapping group string information closest to the center point of the preliminary cropped thumbnail is used as the center point frame;

[0019] Iterate the target visible light string information within a preset distance from the center point frame in order from near to far, and calculate the deviation between a target visible light string information and the center point frame in each iteration;

[0020] Add the deviation to all mapped string information to obtain mobile string information;

[0021] Calculate the first iou of all mobile string information and all visible light string information;

[0022] Taking a deviation as a preliminary optimal value based on the first iou and ending the iteration;

[0023] Control the mobile group string information to move a preset iteration step in an offset direction to obtain offset group string information, wherein the offset direction is a direction from the center of the infrared image to the center of the visible light image;

[0024] After each movement, a second iou of all offset string information and all visible light string information is calculated until the second iou is greater than a first threshold;

[0025] When the second iou is greater than the first threshold, a first movement deviation is calculated according to the current movement times, wherein the first movement deviation=current movement times*preset iteration step length;

[0026] The sum of the initial optimal value and the first movement deviation is taken as the optimal deviation value.

[0027] Further, taking a deviation as a preliminary optimal value based on the first iou includes:

[0028] If the first iou is greater than the second threshold, the current deviation is taken as the preliminary optimal value.

[0029] Further, taking a deviation as a preliminary optimal value based on the first iou includes:

[0030] If the first iou are all smaller than the second threshold, the deviation when the first iou is maximum is taken as the preliminary optimal value.

[0031] Furthermore, it also includes:

[0032] If after the preset number of moves, the second iou is less than the third threshold each time, then obtain N target moving group string information, the target moving group string information is the first N moving group string information in the order from near to far from the preliminary cropped thumbnail, and N is a positive integer;

[0033] Control the target mobile group string information to move a preset iteration step length in the offset direction to obtain the target offset group string information;

[0034] After each movement, a third iou of all target offset string information and all visible light string information is calculated until the third iou is greater than the first threshold;

[0035] When the third iou is greater than the first threshold, a second movement deviation is calculated according to the current movement times, where the second movement deviation=current movement times*preset iteration step length;

[0036] The sum of the initial optimal value and the second movement deviation is taken as the optimal deviation value.

[0037] Furthermore, it also includes:

[0038] If all third iou are not greater than the first threshold, obtain the third iou with the largest value and its corresponding third movement deviation, where the third movement deviation = the corresponding movement number * the preset iteration step, and the corresponding movement number is the movement number corresponding to the third iou with the largest value;

[0039] Respectively obtain the deviation distance from the target mobile group string information whose third iou is greater than a fourth threshold to the center of the cropped thumbnail;

[0040] Obtaining deviations in the x-direction and the y-direction based on the deviation distance;

[0041] If the deviation in the x direction or the y direction is less than a preset distance, it is determined to be a valid deviation;

[0042] Calculate the mean of all valid deviations in the x-direction and y-direction respectively;

[0043] The sum of the initial optimal value, the third moving deviation and the mean value is taken as the optimal deviation value.

[0044] Further, the obtaining of the first distance and the second distance includes:

[0045] Get the first distance and the second distance based on any of the following methods:

[0046] Acquired by laser ranging device;

[0047] Acquired through the altitude and inclination information of each waypoint;

[0048] Modeling the visible light image, acquiring DSM information of the visible light image, and acquiring based on the DSM information.

[0049] Furthermore, before respectively extracting the infrared group string information and the visible light group string information from the infrared image and the preliminary cropped thumbnail, the method further includes:

[0050] De-distortion operations are performed on the infrared image and the initially cropped thumbnail respectively.

[0051] In a second aspect, a photovoltaic inspection image registration device is provided, comprising:

[0052] A shooting distance acquisition module, used to acquire a first distance and a second distance, wherein the first distance is the distance from the shooting point of the infrared image to the shooting target, and the second distance is the distance from the shooting point of the visible light image to the same shooting target, and the content on the visible light image includes the content on the infrared image;

[0053] An initial matrix acquisition module, used to obtain an initial affine matrix based on the first distance and the second distance;

[0054] A visible light image clipping module, used for clipping the visible light image based on the initial affine matrix to obtain a preliminary clipped image with the same content as the infrared image;

[0055] A string extraction module, used to extract infrared string information and visible light string information from the infrared image and the preliminary cropped thumbnail respectively;

[0056] An infrared string mapping module, used to map the infrared string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped string information;

[0057] An optimal deviation acquisition module, used to obtain an optimal deviation value based on the mapping string information and the visible light string information;

[0058] A target matrix acquisition module, used for adjusting the initial affine matrix based on the optimal deviation value to obtain a target affine matrix;

[0059] An image registration module is used to register the infrared image and the visible light image based on the target affine matrix.

[0060] In a third aspect, a photovoltaic inspection image registration system is provided, comprising:

[0061] at least one processor and at least one memory;

[0062] The memory stores executable instructions of the processor;

[0063] The processor is configured to execute the photovoltaic inspection image registration method described above.

[0064] Beneficial effects:

[0065] The technical solution of the present application provides a photovoltaic inspection image registration method, device and system, firstly, an initial affine matrix is ​​obtained according to the distance from the shooting point to the shooting target, then a preliminary cropped small picture with the same content as the infrared image is cropped based on the initial affine matrix, and then the string information on the infrared image and the preliminary cropped small picture is extracted respectively; then the infrared string information is mapped to the preliminary cropped small picture to obtain the mapped string information. An optimal deviation value is obtained based on the deviation between the mapped string information and the visible light string information, and then the initial affine matrix is ​​adjusted based on the optimal deviation value to obtain a target affine matrix, and finally the image is registered based on the target affine matrix. After obtaining the initial affine matrix, the solution of the present application determines the deviation of the initial affine matrix according to the string information on the image, and then adjusts the initial affine matrix based on the deviation, so that the target affine matrix finally obtained is more accurate during image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 This is a flow chart of a photovoltaic inspection image registration method provided in an embodiment of the present application;

[0068] Figure 2is a schematic diagram of capturing a visible light image and an infrared image of the same target provided by an embodiment of the present application;

[0069] Figure 3 is a schematic diagram of image division provided in an embodiment of the present application;

[0070] Figure 4 It is a schematic diagram of the structure of a photovoltaic inspection image registration device provided in an embodiment of the present application;

[0071] Figure 5 It is a schematic diagram of the structure of a photovoltaic inspection image registration system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application is described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other implementation methods obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0073] Hovering photography during photovoltaic inspection can solve the problem of dual-light image fusion, but hovering photography sacrifices inspection efficiency and increases the loss of drones, resulting in increased inspection costs. For the above problems, this solution proposes a registration solution for non-hovering inspections. When the drone is not hovering, due to the interval between dual-light photography, the centers of the two are not coincident, and the offsets are different with different moving directions. Therefore, an affine matrix needs to be calculated for each pair of images. Therefore, this solution first uses the positioning principle to calculate the same-name points of the two to obtain the initial affine matrix, and then uses deep learning to extract the string of dual-light images. Finally, the best matching position is searched according to the moving direction of the camera, and the precise affine matrix of the two is obtained to solve the non-hovering registration problem, thereby realizing non-hovering registration. In this way, the drone can pass through the waypoints at high speed without hovering during the inspection process, thereby realizing rapid inspection. The speed is calculated to be more than 5 times that of hovering, which greatly improves the inspection efficiency and reduces the battery loss of the drone, making the inspection cost lower.

[0074] Reference Figure 1 , the embodiment of the present application provides a photovoltaic inspection image registration method, comprising:

[0075] S11: Acquire a first distance and a second distance, wherein the first distance is the distance from the infrared image shooting point to the shooting target, and the second distance is the distance from the visible light image shooting point to the same shooting target, and the content on the visible light image includes the content on the infrared image.

[0076] When laser ranging is available, some drone inspection models are equipped with laser ranging devices (more expensive), and the first distance and the second distance are obtained through the laser ranging device.

[0077] When there is no laser ranging but there is a route, since the altitude and inclination information of each waypoint is recorded during route planning, the target distance from the drone to the waypoint when taking pictures can be calculated, that is, the first distance and the second distance.

[0078] When there is no laser ranging and route, due to the large field of view of visible light and the corresponding large overlap, in this case, we model the collected visible light image and calculate the DSM (Digital Surface Model) information of the image, so that we can obtain the first distance and the second distance.

[0079] S12: Obtaining an initial affine matrix based on the first distance and the second distance;

[0080] Specifically, the initial affine matrix is ​​calculated using the positioning principle to achieve rough configuration. The mapping of geographic coordinates to pixel coordinates can be achieved by combining the intrinsic parameters of infrared and visible light cameras, as well as the extrinsic parameters during shooting (pitch angle during shooting) and shooting distance. The positioning principle is the conversion of pixel coordinates of the image captured by the camera to the real geographic coordinates, which is a common technology in the field and will not be described in detail here.

[0081] It should be noted here that when shooting with dual light, the field of view of visible light is larger than that of infrared, and the content of the infrared image must be within the content of the visible light, such as Figure 2 As shown, the large rectangle outside is the visible light image, the small rectangle inside is the infrared image, and the two points are the center of the visible light image and the center of the infrared image respectively. With this prerequisite, we can calculate the same-name points of the two, because the two images must have been taken at the same position, so we can select some points in the common area and calculate the pixel coordinates of the two respectively, so that we can calculate the affine change from infrared to visible light.

[0082] Consider that visible light contains infrared, such as Figure 3As shown, we evenly select several points on the infrared (it must be even or it will affect the registration), take the center point as the original point, divide the image into 4 blocks and then calculate the center point of each block, so that the 5 center points obtained are used as the basis to generate random and evenly distributed points (that is, the small rectangle is divided into four smaller rectangles by the midpoint of its length and width, and then the center of the smaller rectangle is used as the selected point. The specific number of divisions is determined according to actual needs. The more points are actually used, the more accurate the initial affine matrix is, but the relative amount of calculation is larger). Then, the positioning principle and infrared camera parameters are used to calculate its real geographic coordinates, and then the pixel coordinates on the visible light are calculated based on these geographic coordinates and visible light parameters. In this way, the correspondence between the infrared points and the visible light points can be calculated, and the initial affine matrix can be obtained (there is a deviation in positioning, so the initial matrix also has some deviation).

[0083] In S11-S12, we calculated the initial affine matrix. Therefore, for each pair of images, we first cropped the infrared in the visible light and recorded it as the initial cropped image. Then we dedistorted the image, and then segmented the infrared image and the initial cropped image to extract the string information. We then used the affine matrix to map the infrared string coordinates to the visible light so that they can be matched in the same coordinate system.

[0084] The string information is extracted by using a pre-trained model, for example:

[0085] Build your own string segmentation dataset. The images come from dual-light images collected by photovoltaic stations. Separate the infrared and visible light data and build a dataset for each.

[0086] The original size of infrared images is 640*512, and the image does not need to be cropped. However, the size of visible light is about 4000*3000, and the segmentation needs to be scaled. Therefore, we first preprocess the visible light and extract the infrared area on the visible light through a large number of statistics. The long side resolution is about 1500-2000, and the short side is between 1300-1600. Therefore, for the visible light data, random cropping is performed according to the above resolution prior knowledge to generate several small images. In this way, the field of view in the visible light is as consistent as possible with that in the infrared, providing a better segmentation model for the subsequent rough registration and preliminary cropping of small images.

[0087] Due to the distortion problem in camera shooting, the distortion will affect the effect of group string recognition. Here we will dedistort the image to make the group string more normal, so as to better extract the group string. At the same time, it should be noted that our annotations are marked on the original image. During training, we will distort the annotations and the original image to obtain new training data. In addition, we will extract some original images and insert them into the new data to indirectly achieve data enhancement effect, so as to better realize group string extraction.

[0088] For infrared and visible light data, two models are trained separately. Here, yolov8 is selected as the segmentation model, and the input size is uniformly resized to 640 to achieve fast segmentation. Then the segmentation result is used to fine-tune the registration result to achieve precise registration. The details are as follows:

[0089] S13: Cropping the visible light image based on the initial affine matrix to obtain a preliminary cropped thumbnail with the same content as the infrared image.

[0090] S14: extracting infrared group string information and visible light group string information from the infrared image and the preliminary cropped thumbnail respectively.

[0091] Preferably, before respectively extracting the infrared group string information and the visible light group string information from the infrared image and the preliminary cropped small image, the method further comprises: performing a dedistortion operation on the infrared image and the preliminary cropped small image respectively.

[0092] S15: Mapping the infrared group string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped group string information.

[0093] S16: Obtaining an optimal deviation value based on the mapping string information and the visible light string information;

[0094] As a preferred implementation of the embodiment of the present application, obtaining the optimal deviation value based on the mapping string information and the visible light string information includes:

[0095] The mapping string information is sorted according to the distance from the mapping string information to the center point of the preliminary cropped small image; the border of the mapping string information closest to the center point of the preliminary cropped small image is used as the center point frame; the target visible light string information within the preset distance to the center point frame is iterated in order from near to far, and each iteration calculates the deviation of a target visible light string information and the center point frame; all mapping string information are added with the deviation to obtain mobile string information; the first iou of all mobile string information and all visible light string information is calculated; based on the first iou, a The deviation is taken as the preliminary optimal value and the iteration is ended; the mobile string information is controlled to move in the offset direction by a preset iteration step to obtain the offset string information, and the offset direction is the direction from the center of the infrared image to the center of the visible light image; after each movement, the second iou of all the offset string information and all the visible light string information is calculated until the second iou is greater than the first threshold; when the second iou is greater than the first threshold, the first movement deviation is calculated according to the current number of movements, and the first movement deviation = the current number of movements * the preset iteration step; the sum of the initial optimal value and the first movement deviation is taken as the optimal deviation value.

[0096] Wherein, taking a deviation as a preliminary optimal value based on the first iou comprises:

[0097] If the first iou is greater than the second threshold, the current deviation is used as the preliminary optimal value. If the first iou are all less than the second threshold, the deviation when the first iou is the largest is used as the preliminary optimal value.

[0098] In addition, if after moving a preset number of times, the second iou is less than the third threshold each time, then obtain N target mobile group string information, the target mobile group string information is the first N mobile group string information in the order from near to far from the preliminary cropped thumbnail, and N is a positive integer; control the target mobile group string information to move the preset iteration step in the offset direction to obtain the target offset group string information; calculate the third iou of all target offset group string information and all visible light group string information after each movement until the third iou is greater than the first threshold; when the third iou is greater than the first threshold, calculate the second movement deviation according to the current number of movements, the second movement deviation = the current number of movements * the preset iteration step; and take the sum of the initial optimal value and the second movement deviation as the optimal deviation value.

[0099] If all third iou are not greater than the first threshold, obtain the third iou with the largest value and its corresponding third moving deviation, the third moving deviation = corresponding number of moves * preset iteration step, the corresponding number of moves is the number of moves corresponding to the third iou with the largest value; obtain the deviation distance from the target moving group string information of the third iou greater than the fourth threshold to the center of the cropped thumbnail respectively; obtain the deviation in the x-direction and y-direction based on the deviation distance; if the deviation in the x-direction or y-direction is less than the preset distance, it is determined to be a valid deviation; respectively calculate the average of all valid deviations in the x-direction and y-direction; and take the sum of the initial optimal value, the third moving deviation and the average as the optimal deviation value.

[0100] Exemplarily, a. Sort the extracted infrared and visible light string information according to the distance from the center point of the cropped image, fix the infrared string closest to the center point frame (denoted as ir1), find all visible light strings whose distance from ir1 (the distance between the center points of the two) is within a certain range (such as 200 pixels), calculate the deviation between the two, and then add the deviation to all infrared strings, calculate the iou of the two, if the iou is greater than 0.8, stop the iteration, that is, the deviation is the preliminary optimal deviation value, otherwise continue the iteration, if all do not meet the requirements, take the deviation with the largest iou as the preliminary optimal value.

[0101] b. Since the coordinates of the infrared and visible light shooting points are recorded during shooting, the relative positions of the two are known, that is, the approximate direction of the relative offset is known. Then, the iteration step is set according to the offset direction (such as one step for every 5 pixels, and the maximum iteration is terminated at 200 pixels), the optimal matching path is found, and the optimal matching iou is calculated. If the iou is greater than 0.95, the iteration is terminated, and the result is the optimal deviation. Otherwise, if the iou is less than 0.6, take the first 6 strings closest to the center (if there are less than 6, take as many as you want), and repeat the above operation again. This solution avoids interference caused by too many strings at the boundaries.

[0102] c. If the iou stopping iteration threshold is still not reached, this is usually because the clusters are close to the edge and there are no clusters in the center. Due to segmentation and distortion, some deviations are likely to occur, and the deviation between the two needs to be optimized. Based on the above movement, the first six pairs of matching clusters are corrected (may be less than 6 pairs). If the iou between the two is greater than 0.5, they are corrected. The distance from each to the center of the cropped image is calculated to obtain the deviation in the x and y directions. At the same time, the deviation is restricted (such as not exceeding 50 pixels). If it exceeds the limit, it is recorded as an invalid deviation. All valid deviations are recorded, and the average is finally calculated to obtain the deviation adjustment of the edge. The final deviation is returned, so that images with only clusters at the boundary can also be well registered.

[0103] If the iteration stops at step b, step c does not need to be executed. The idea of ​​registration is to ensure that the pairs close to the center are registered as much as possible. There are some deviations at the boundaries due to distortion and other reasons. Therefore, step c is added so that the target affine matrix can be calculated for each pair of non-hovering images.

[0104] S17: Adjust the initial affine matrix based on the optimal deviation value to obtain a target affine matrix.

[0105] S18: Registering the infrared image and the visible light image based on the target affine matrix.

[0106] The photovoltaic inspection image registration method provided in the embodiment of the present application first obtains an initial affine matrix based on the distance from the shooting point to the shooting target, and then crops a preliminary cropped thumbnail with the same content as the infrared image based on the initial affine matrix, and then extracts the string information on the infrared image and the preliminary cropped thumbnail respectively; then the infrared string information is mapped to the preliminary cropped thumbnail to obtain the mapped string information. An optimal deviation value is obtained based on the deviation between the mapped string information and the visible light string information, and then the initial affine matrix is ​​adjusted based on the optimal deviation value to obtain a target affine matrix, and finally the image is registered based on the target affine matrix. After obtaining the initial affine matrix, the present application scheme determines the deviation of the initial affine matrix based on the string information on the image, and then adjusts the initial affine matrix based on the deviation, so that the target affine matrix finally obtained is more accurate during image registration.

[0107] Based on the same inventive concept, Figure 4 As shown, the present application also provides a photovoltaic inspection image registration device 40, comprising:

[0108] The shooting distance acquisition module 41 is used to acquire a first distance and a second distance, wherein the first distance is the distance from the shooting point of the infrared image to the shooting target, and the second distance is the distance from the shooting point of the visible light image to the same shooting target, and the content on the visible light image includes the content on the infrared image;

[0109] The first distance and the second distance are obtained based on any of the following methods:

[0110] Acquired by laser ranging device;

[0111] Acquired through the altitude and inclination information of each waypoint;

[0112] Modeling the visible light image, acquiring DSM information of the visible light image, and acquiring based on the DSM information.

[0113] The initial matrix acquisition module 42 is used to obtain an initial affine matrix based on the first distance and the second distance.

[0114] The visible light image clipping module 43 is used to clip the visible light image based on the initial affine matrix to obtain a preliminary clipped image with the same content as the infrared image.

[0115] The string extraction module 44 is used to extract infrared string information and visible light string information from the infrared image and the preliminary cropped thumbnail respectively.

[0116] It should be noted that before respectively extracting the infrared group string information and the visible light group string information from the infrared image and the preliminary cropped small image, the method further includes: performing a dedistortion operation on the infrared image and the preliminary cropped small image respectively.

[0117] The infrared string mapping module 45 is used to map the infrared string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped string information.

[0118] The optimal deviation acquisition module 46 is used to obtain an optimal deviation value based on the mapping string information and the visible light string information.

[0119] As a preferred implementation method of the embodiment of the present application: the obtaining of the optimal deviation value based on the mapping string information and the visible light string information includes:

[0120] The mapping string information is sorted according to the distance from the mapping string information to the center point of the preliminary cropped small image; the border of the mapping string information closest to the center point of the preliminary cropped small image is used as the center point frame; the target visible light string information within the preset distance to the center point frame is iterated in order from near to far, and each iteration calculates the deviation of a target visible light string information and the center point frame; all mapping string information are added with the deviation to obtain mobile string information; the first iou of all mobile string information and all visible light string information is calculated; based on the first iou, a The deviation is taken as the preliminary optimal value and the iteration is ended; the mobile string information is controlled to move in the offset direction by a preset iteration step to obtain the offset string information, and the offset direction is the direction from the center of the infrared image to the center of the visible light image; after each movement, the second iou of all the offset string information and all the visible light string information is calculated until the second iou is greater than the first threshold; when the second iou is greater than the first threshold, the first movement deviation is calculated according to the current number of movements, and the first movement deviation = the current number of movements * the preset iteration step; the sum of the initial optimal value and the first movement deviation is taken as the optimal deviation value.

[0121] The taking a deviation as a preliminary optimal value based on the first iou comprises:

[0122] If the first iou is greater than the second threshold, the current deviation is used as the preliminary optimal value. If the first iou are all less than the second threshold, the deviation when the first iou is the largest is used as the preliminary optimal value.

[0123] In addition, if after moving a preset number of times, the second iou is less than the third threshold each time, then obtain N target mobile group string information, the target mobile group string information is the first N mobile group string information in the order from near to far from the preliminary cropped thumbnail, and N is a positive integer; control the target mobile group string information to move the preset iteration step in the offset direction to obtain the target offset group string information; calculate the third iou of all target offset group string information and all visible light group string information after each movement until the third iou is greater than the first threshold; when the third iou is greater than the first threshold, calculate the second movement deviation according to the current number of movements, the second movement deviation = the current number of movements * the preset iteration step; and take the sum of the initial optimal value and the second movement deviation as the optimal deviation value.

[0124] Further, if all third iou are not greater than the first threshold, obtain the third iou with the largest value and its corresponding third moving deviation, the third moving deviation = corresponding number of moves * preset iteration step, the corresponding number of moves is the number of moves corresponding to the third iou when the value is the largest; obtain the deviation distance from the target moving group string information whose third iou is greater than the fourth threshold to the center of the cropped thumbnail respectively; obtain the deviation in the x-direction and y-direction based on the deviation distance; if the deviation in the x-direction or y-direction is less than the preset distance, it is determined to be a valid deviation; respectively calculate the mean of all valid deviations in the x-direction and y-direction; and take the sum of the initial optimal value, the third moving deviation and the mean as the optimal deviation value.

[0125] The target matrix acquisition module 47 is used to adjust the initial affine matrix based on the optimal deviation value to obtain a target affine matrix.

[0126] The image registration module 48 is used to register the infrared image and the visible light image based on the target affine matrix.

[0127] The photovoltaic inspection image registration device provided in the embodiment of the present application first obtains an initial affine matrix based on the distance from the shooting point to the shooting target, and then crops a preliminary cropped thumbnail with the same content as the infrared image based on the initial affine matrix, and then extracts the string information on the infrared image and the preliminary cropped thumbnail respectively; then the infrared string information is mapped to the preliminary cropped thumbnail to obtain the mapped string information. An optimal deviation value is obtained based on the deviation between the mapped string information and the visible light string information, and then the initial affine matrix is ​​adjusted based on the optimal deviation value to obtain a target affine matrix, and finally image registration is performed based on the target affine matrix. After obtaining the initial affine matrix, the present application solution determines the deviation of the initial affine matrix based on the string information on the image, and then adjusts the initial affine matrix based on the deviation, so that the target affine matrix finally obtained is more accurate during image registration.

[0128] Based on the same inventive concept, Figure 5 As shown, the present application also provides a photovoltaic inspection image registration system 50, comprising:

[0129] at least one processor 51 and at least one memory 52;

[0130] The memory stores executable instructions of the processor;

[0131] The processor is configured to execute the photovoltaic inspection image registration method provided in the above embodiment.

[0132] The photovoltaic inspection image registration system provided by the embodiment of the present application stores the executable instructions of the processor in the memory. When the executable instructions are executed, the processor can first obtain the initial affine matrix according to the distance from the shooting point to the shooting target, and then crop the preliminary cropped small picture with the same content as the infrared image based on the initial affine matrix, and then extract the string information on the infrared image and the preliminary cropped small picture respectively; then map the infrared string information to the preliminary cropped small picture to obtain the mapped string information. The optimal deviation value is obtained based on the deviation between the mapped string information and the visible light string information, and then the initial affine matrix is ​​adjusted based on the optimal deviation value to obtain the target affine matrix, and finally the image is registered according to the target affine matrix. After obtaining the initial affine matrix, the present application solution determines the deviation of the initial affine matrix according to the string information on the image, and then adjusts the initial affine matrix based on the deviation, so that the target affine matrix finally obtained is more accurate in image registration.

[0133] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0134] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

Claims

1. A photovoltaic inspection image registration method, characterized in that: include: Acquire a first distance and a second distance, wherein the first distance is the distance from the infrared image shooting point to the shooting target, and the second distance is the distance from the visible light image shooting point to the same shooting target, and the content on the visible light image includes the content on the infrared image; Obtaining an initial affine matrix based on the first distance and the second distance; Cropping the visible light image based on the initial affine matrix to obtain a preliminary cropped thumbnail with the same content as the infrared image; Extracting infrared group string information and visible light group string information from the infrared image and the preliminary cropped thumbnail respectively; Mapping the infrared group string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped group string information; Obtaining an optimal deviation value based on the mapping string information and the visible light string information; Adjusting the initial affine matrix based on the optimal deviation value to obtain a target affine matrix; The infrared image and the visible light image are registered based on the target affine matrix.

2. The method according to claim 1, characterized in that: The obtaining the optimal deviation value based on the mapping string information and the visible light string information includes: Sorting the mapping group string information according to the distance from the mapping group string information to the center point of the preliminary cropped thumbnail; The frame of the mapping group string information closest to the center point of the preliminary cropped thumbnail is used as the center point frame; Iterate the target visible light string information within a preset distance from the center point frame in order from near to far, and calculate the deviation between a target visible light string information and the center point frame in each iteration; Add the deviation to all mapped string information to obtain mobile string information; Calculate the first iou of all mobile string information and all visible light string information; Taking a deviation as a preliminary optimal value based on the first iou and ending the iteration; Control the mobile group string information to move a preset iteration step in an offset direction to obtain offset group string information, wherein the offset direction is a direction from the center of the infrared image to the center of the visible light image; After each movement, a second iou of all offset string information and all visible light string information is calculated until the second iou is greater than a first threshold; When the second iou is greater than the first threshold, a first movement deviation is calculated according to the current movement times, wherein the first movement deviation=current movement times*preset iteration step length; The sum of the initial optimal value and the first movement deviation is taken as the optimal deviation value.

3. The method according to claim 2, characterized in that: The taking a deviation as a preliminary optimal value based on the first iou comprises: If the first iou is greater than the second threshold, the current deviation is taken as the preliminary optimal value.

4. The method according to claim 2, characterized in that: The taking a deviation as a preliminary optimal value based on the first iou comprises: If the first iou are all smaller than the second threshold, the deviation when the first iou is maximum is taken as the preliminary optimal value.

5. The method according to claim 2, characterized in that: Also includes: If after the preset number of moves, the second iou is less than the third threshold each time, then obtain N target moving group string information, the target moving group string information is the first N moving group string information in the order from near to far from the preliminary cropped thumbnail, and N is a positive integer; Control the target mobile group string information to move a preset iteration step length in the offset direction to obtain the target offset group string information; After each movement, a third iou of all target offset string information and all visible light string information is calculated until the third iou is greater than the first threshold; When the third iou is greater than the first threshold, a second movement deviation is calculated according to the current movement times, where the second movement deviation=current movement times*preset iteration step length; The sum of the initial optimal value and the second movement deviation is taken as the optimal deviation value.

6. The method according to claim 5, characterized in that Also includes: If all third iou are not greater than the first threshold, obtain the third iou with the largest value and its corresponding third movement deviation, where the third movement deviation = the corresponding movement number * the preset iteration step, and the corresponding movement number is the movement number corresponding to the third iou with the largest value; Respectively obtain the deviation distance from the target mobile group string information whose third iou is greater than a fourth threshold to the center of the cropped thumbnail; Obtaining deviations in the x-direction and the y-direction based on the deviation distance; If the deviation in the x direction or the y direction is less than a preset distance, it is determined to be a valid deviation; Calculate the mean of all valid deviations in the x-direction and y-direction respectively; The sum of the initial optimal value, the third moving deviation and the mean value is taken as the optimal deviation value.

7. The method according to claim 1, characterized in that: The obtaining of the first distance and the second distance comprises: Get the first distance and the second distance based on any of the following methods: Acquired by laser ranging device; Acquired through the altitude and inclination information of each waypoint; Modeling the visible light image, acquiring DSM information of the visible light image, and acquiring based on the DSM information.

8. The method according to claim 1, characterized in that: Before respectively extracting the infrared group string information and the visible light group string information from the infrared image and the preliminary cropped thumbnail, the method further includes: De-distortion operations are performed on the infrared image and the initially cropped thumbnail respectively.

9. A photovoltaic inspection image registration device, characterized in that: include: A shooting distance acquisition module, used to acquire a first distance and a second distance, wherein the first distance is the distance from the shooting point of the infrared image to the shooting target, and the second distance is the distance from the shooting point of the visible light image to the same shooting target, and the content on the visible light image includes the content on the infrared image; An initial matrix acquisition module, used to obtain an initial affine matrix based on the first distance and the second distance; A visible light image clipping module, used for clipping the visible light image based on the initial affine matrix to obtain a preliminary clipped image with the same content as the infrared image; A string extraction module, used to extract infrared string information and visible light string information from the infrared image and the preliminary cropped thumbnail respectively; An infrared string mapping module, used to map the infrared string information onto the preliminary cropped thumbnail based on the initial affine matrix to obtain mapped string information; An optimal deviation acquisition module, used to obtain an optimal deviation value based on the mapping string information and the visible light string information; A target matrix acquisition module, used for adjusting the initial affine matrix based on the optimal deviation value to obtain a target affine matrix; An image registration module is used to register the infrared image and the visible light image based on the target affine matrix.

10. A photovoltaic inspection image registration system, characterized in that: include: at least one processor and at least one memory; The memory stores executable instructions of the processor; The processor is configured to execute the method according to any one of claims 1 to 8.

Images