Image matching method, device, equipment, storage medium and product

Abstract

This application belongs to the field of image processing technology and discloses an image matching method, apparatus, device, storage medium, and product. This application acquires infrared and visible light images of a photovoltaic inspection area, then performs string segmentation on both the infrared and visible light images to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image. Next, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings to obtain a registration relationship set. Finally, it performs image matching on the infrared and visible light images based on the registration relationship set. This application can perform pixel semantic segmentation on both infrared and visible light images to obtain multiple infrared and visible light strings, then perform point cloud registration using the infrared and visible light point cloud data sets, and finally achieve image matching between the infrared and visible light images based on the registration relationship set.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image matching method, apparatus, device, storage medium and product. Background Technology

[0002] With the continuous expansion of energy and electricity demand, traditional manual inspection methods are insufficient to meet the operation and maintenance needs of large-scale photovoltaic power plants, urgently requiring more efficient and accurate inspection methods. Drone inspection technology, as a rapid, non-contact inspection method, can quickly and comprehensively cover all areas of a photovoltaic power plant, significantly shortening inspection time. Photovoltaic drone inspection uses drones equipped with infrared and visible light lenses to capture images of the entire photovoltaic area, and further uses intelligent diagnostic algorithms to identify various faults that may occur in photovoltaic modules during operation, such as hot spots, open circuits, and short circuits. Infrared images can reflect the temperature distribution of objects, making them very effective for detecting hot spots, non-generating strings, junction boxes, and other faults in photovoltaic modules. Visible light images have high spatial resolution and can provide appearance information of photovoltaic modules, such as their shape, color, and texture. They have excellent imaging effects for missing, broken, or obstructed components. Therefore, the two imaging methods are usually combined in photovoltaic inspection. Due to differences in lens and sensor characteristics, shooting angles, and positions between infrared and visible light images from drones, variations in viewpoint, image size, and quality occur when photographing the same object, leading to difficulties in infrared and visible light registration. Therefore, effectively matching infrared and visible light images has become a pressing problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide an image matching method, apparatus, device, storage medium, and product, which aims to solve the technical problem of how to effectively match infrared images and visible light images.

[0004] To achieve the above objectives, this application provides an image matching method, which includes the following steps:

[0005] Collect infrared and visible light images of the photovoltaic inspection area;

[0006] The infrared image and the visible light image are respectively segmented into multiple infrared strings in the infrared image and multiple visible light strings in the visible light image;

[0007] Register the infrared point cloud data set corresponding to the plurality of infrared strings and the visible light point cloud data set corresponding to the plurality of visible light strings to obtain a registration relationship set;

[0008] Image matching is performed on the infrared image and the visible light image based on the registration relationship set.

[0009] Optionally, the step of registering the infrared point cloud data set corresponding to the plurality of infrared strings and the visible light point cloud data set corresponding to the plurality of visible light strings to obtain a registration relationship set specifically includes:

[0010] The infrared point cloud data set is determined based on the set of infrared pixel coordinates corresponding to the multiple infrared strings;

[0011] The visible light point cloud data set is determined based on the set of visible light pixel coordinates corresponding to the multiple visible light strings;

[0012] The infrared point cloud data set and the visible light point cloud data set are registered to obtain a set of registration relationships.

[0013] Optionally, the step of determining the infrared point cloud data set based on the set of infrared pixel coordinates corresponding to the plurality of infrared strings specifically includes:

[0014] The infrared image is reconstructed in three dimensions to obtain the reconstructed infrared point cloud data;

[0015] The reconstructed infrared point cloud data is adjusted to obtain initial infrared point cloud data;

[0016] Construct an infrared mapping relationship between the initial infrared pixel coordinates of all infrared pixels in the infrared image and the initial infrared point cloud data;

[0017] The infrared point cloud data set is determined based on the set of infrared pixel coordinates corresponding to the multiple infrared strings and the infrared mapping relationship.

[0018] Optionally, the step of adjusting the reconstructed infrared point cloud data to obtain initial infrared point cloud data specifically includes:

[0019] The target Z-axis coordinates of all infrared pixels in the infrared image are determined based on the initial Z-axis coordinates in the reconstructed infrared point cloud data.

[0020] A first relation is constructed based on the initial infrared pixel coordinates of all infrared pixels in the infrared image, the camera intrinsic parameter matrix, and the camera extrinsic parameter matrix.

[0021] The camera projection matrix is ​​determined based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and the first relation is converted into a second relation based on the camera projection matrix;

[0022] The initial infrared point cloud data corresponding to all infrared pixels in the infrared image is determined based on the target Z-axis coordinates and the second relational formula.

[0023] Optionally, the step of registering the infrared point cloud data set and the visible light point cloud data set to obtain a registration relationship set specifically includes:

[0024] Select a target infrared string from the plurality of infrared strings, and select a target visible light string from the plurality of visible light strings;

[0025] Select the target infrared point cloud data corresponding to the target infrared string from the infrared point cloud data set, and select the target visible light point cloud data corresponding to the target visible light string from the visible light point cloud data set;

[0026] The infrared point cloud data and the visible light point cloud data of the target are registered to obtain the registration relationship;

[0027] Return to the step of selecting the target infrared string from the plurality of infrared strings, obtain a new registration relationship, and construct a registration relationship set.

[0028] Optionally, the step of registering the target infrared point cloud data and the target visible light point cloud data to obtain a registration relationship specifically includes:

[0029] The target infrared point cloud data and the target visible light point cloud data are registered based on the iterative nearest point algorithm to obtain the registration relationship, which includes the mapping relationship between the target infrared point cloud data and the target visible light point cloud data.

[0030] Optionally, the step of performing image matching between the infrared image and the visible light image based on the registration relationship set specifically includes:

[0031] Determine all registration relationships in the registration relationship set;

[0032] Determine the registration infrared point cloud data and registration visible light point cloud data in each registration relationship;

[0033] Image matching is performed on the infrared image and the visible light image based on the registered infrared point cloud data and the registered visible light point cloud data.

[0034] Optionally, the step of performing image matching between the infrared image and the visible light image based on the registered infrared point cloud data and the registered visible light point cloud data specifically includes:

[0035] The coordinates of the registered infrared pixel points are determined based on the registered infrared point cloud data and the infrared mapping relationship.

[0036] The coordinates of the registered visible light pixels are determined based on the registered visible light point cloud data and the visible light mapping relationship.

[0037] Image matching is performed on the infrared image and the visible light image based on the registered infrared pixel coordinates and the registered visible light pixel coordinates.

[0038] Furthermore, to achieve the above objectives, this application also provides an image matching device, the image matching device comprising:

[0039] The image acquisition module is used to acquire infrared and visible light images of the photovoltaic inspection area;

[0040] The string segmentation module is used to perform string segmentation on the infrared image and the visible light image respectively, to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image;

[0041] The data registration module is used to register the infrared point cloud data set corresponding to the multiple infrared strings and the visible light point cloud data set corresponding to the multiple visible light strings to obtain a registration relationship set.

[0042] An image matching module is used to perform image matching between the infrared image and the visible light image based on the registration relationship set.

[0043] In addition, to achieve the above objectives, this application also proposes an image matching device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image matching method as described above.

[0044] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the image matching method described above.

[0045] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image matching method described above.

[0046] This application acquires infrared and visible light images of a photovoltaic inspection area, then performs string segmentation on both infrared and visible light images to obtain multiple infrared strings and visible light strings respectively. Next, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings to obtain a registration relationship set. Finally, it performs image matching on the infrared and visible light images based on the registration relationship set. This application first performs string segmentation on both infrared and visible light images, enabling pixel semantic segmentation to obtain multiple infrared and visible light strings. Then, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings, enabling point cloud registration using the infrared and visible light point cloud data sets. Finally, it achieves image matching between the infrared and visible light images based on the registration relationship set. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating the first embodiment of the image matching method of this application;

[0050] Figure 2 This is a schematic diagram of an infrared image and a visible light image, representing an embodiment of the image matching method of this application.

[0051] Figure 3 This is a schematic diagram of an infrared string and a visible light string according to an embodiment of the image matching method of this application;

[0052] Figure 4 This is a flowchart illustrating the second embodiment of the image matching method of this application;

[0053] Figure 5 This is a schematic diagram of target infrared point cloud data and target visible light point cloud data according to an embodiment of the image matching method of this application;

[0054] Figure 6 This is a schematic diagram of point cloud data registration according to an embodiment of the image matching method of this application;

[0055] Figure 7This is a flowchart illustrating the third embodiment of the image matching method of this application;

[0056] Figure 8 This is a schematic diagram illustrating the matching of infrared and visible light images according to an embodiment of the image matching method of this application;

[0057] Figure 9 This is a structural block diagram of the first embodiment of the image matching device of this application;

[0058] Figure 10 This is a schematic diagram of the structure of the image matching device in the hardware operating environment involved in the embodiments of this application.

[0059] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0061] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The main solution of this application embodiment is as follows: acquiring infrared and visible light images of the photovoltaic inspection area; performing string segmentation on the infrared and visible light images respectively to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image; registering the infrared point cloud data set corresponding to the multiple infrared strings and the visible light point cloud data set corresponding to the multiple visible light strings to obtain a registration relationship set; and performing image matching on the infrared and visible light images according to the registration relationship set.

[0063] With the continuous expansion of energy and electricity demand, traditional manual inspection methods are insufficient to meet the operation and maintenance needs of large-scale photovoltaic power plants, urgently requiring more efficient and accurate inspection methods. Drone inspection technology, as a rapid, non-contact inspection method, can quickly and comprehensively cover all areas of a photovoltaic power plant, significantly shortening inspection time. Photovoltaic drone inspection uses drones equipped with infrared and visible light lenses to capture images of the entire photovoltaic area, and further uses intelligent diagnostic algorithms to identify various faults that may occur in photovoltaic modules during operation, such as hot spots, open circuits, and short circuits. Infrared images can reflect the temperature distribution of objects, making them very effective for detecting hot spots, non-generating strings, junction boxes, and other faults in photovoltaic modules. Visible light images have high spatial resolution and can provide appearance information of photovoltaic modules, such as their shape, color, and texture. They have excellent imaging effects for missing, broken, or obstructed components. Therefore, the two imaging methods are usually combined in photovoltaic inspections. Due to differences in lens, sensor characteristics, shooting angle, and position between infrared and visible light from drones, there are differences in the viewing angle, image size, and quality when shooting the same object, which leads to difficulties in infrared and visible light registration.

[0064] This application acquires infrared and visible light images of a photovoltaic inspection area, then performs string segmentation on both infrared and visible light images to obtain multiple infrared strings and visible light strings respectively. Next, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings to obtain a registration relationship set. Finally, it performs image matching on the infrared and visible light images based on the registration relationship set. This application first performs string segmentation on both infrared and visible light images, enabling pixel semantic segmentation to obtain multiple infrared and visible light strings. Then, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings, enabling point cloud registration using the infrared and visible light point cloud data sets. Finally, it achieves image matching between the infrared and visible light images based on the registration relationship set.

[0065] It should be noted that the executing entity of this application can be a computing service device with data processing, network communication, and program execution functions, such as a computer, or an electronic device or image matching device capable of performing the above functions. The following description uses an image matching device as an example to illustrate this embodiment and the subsequent embodiments.

[0066] Based on this, embodiments of this application provide an image matching method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the image matching method of this application.

[0067] In this embodiment, the image matching method includes the following steps:

[0068] Step S10: Collect infrared and visible light images of the photovoltaic inspection area.

[0069] Understandably, the photovoltaic inspection area refers to the area of ​​the photovoltaic power station that needs to be inspected. By using drones equipped with infrared and visible light cameras, and planning flight routes within the photovoltaic inspection area, and by setting parameters such as flight altitude, heading, and lateral overlap, infrared and visible light images of the photovoltaic area can be collected. Figure 2 , Figure 2 This is a schematic diagram of an infrared image and a visible light image according to an embodiment of the image matching method of this application. The left side is the infrared image and the right side is the visible light image.

[0070] Step S20: Perform string segmentation on the infrared image and the visible light image respectively to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image.

[0071] It should be understood that image semantic segmentation algorithms can be used to perform string segmentation on infrared and visible light images respectively, i.e., pixel-by-pixel semantic segmentation, to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image, as shown below. Figure 3 , Figure 3 This is a schematic diagram of an infrared string and a visible light string according to an embodiment of the image matching method of this application. The left side is the infrared string and the right side is the visible light string.

[0072] Step S30: Register the infrared point cloud data set corresponding to the multiple infrared strings and the visible light point cloud data set corresponding to the multiple visible light strings to obtain a registration relationship set.

[0073] Understandably, multiple infrared point cloud data sets corresponding to infrared strings can be obtained, and the infrared point cloud data sets may include the point cloud data corresponding to each infrared string. Additionally, multiple visible light point cloud data sets corresponding to visible light strings can be obtained, and the visible light point cloud data sets may include the point cloud data corresponding to each visible light string.

[0074] It should be understood that the infrared point cloud dataset and the visible light point cloud dataset can be registered. Specifically, a set of infrared point cloud data can be selected from the infrared point cloud dataset, and a set of visible light point cloud data can be selected from the visible light point cloud dataset. The infrared point cloud data and the visible light point cloud data can then be registered to obtain the registration relationship. Other sets of infrared point cloud data and visible light point cloud data can also be selected to form a set of registration relationships.

[0075] Step S40: Perform image matching on the infrared image and the visible light image according to the registration relationship set.

[0076] In practical implementation, infrared and visible light images can be matched according to the registration relationship set. Specifically, the infrared pixel coordinates corresponding to the registered infrared point cloud data and the visible light pixel coordinates corresponding to the visible light point cloud data can be determined. That is, after registering the infrared point cloud data and the visible light point cloud data, they are mapped to the image pixels to achieve dual-light matching of infrared and visible light images.

[0077] This embodiment acquires infrared and visible light images of a photovoltaic inspection area, then performs string segmentation on both infrared and visible light images to obtain multiple infrared strings and visible light strings respectively. Next, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings to obtain a registration relationship set. Finally, it performs image matching on the infrared and visible light images based on the registration relationship set. This embodiment first performs string segmentation on the infrared and visible light images, enabling pixel semantic segmentation of both images to obtain multiple infrared and visible light strings. Then, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings, enabling point cloud registration using these data sets. Finally, it achieves image matching between the infrared and visible light images based on the registration relationship set.

[0078] refer to Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the image matching method of this application.

[0079] Based on the first embodiment described above, in this embodiment, step S30 includes:

[0080] Step S301: Determine the infrared point cloud data set based on the set of infrared pixel coordinates corresponding to the multiple infrared strings.

[0081] Understandably, each infrared string can correspond to a set of infrared pixel coordinates, thus obtaining a set of infrared pixel coordinates corresponding to multiple infrared strings, and determining the infrared point cloud data set based on the set of infrared pixel coordinates.

[0082] Furthermore, in order to accurately obtain the infrared point cloud data set, in this embodiment, step S301 includes: performing three-dimensional reconstruction on the infrared image to obtain reconstructed infrared point cloud data; adjusting the reconstructed infrared point cloud data to obtain initial infrared point cloud data; constructing an infrared mapping relationship between the initial infrared pixel coordinates of all infrared pixels in the infrared image and the initial infrared point cloud data; and determining the infrared point cloud data set based on the set of infrared pixel coordinates corresponding to the multiple infrared strings and the infrared mapping relationship.

[0083] It should be understood that Structure from Motion (SFM) 3D reconstruction can be performed on infrared images to recover the 3D sparse point cloud of the infrared image. After SFM 3D reconstruction, reconstructed infrared point cloud data corresponding to some pixels in the infrared image can be obtained. Due to the sparse nature of the 3D reconstructed point cloud, that is, not every pixel in the infrared image can obtain corresponding point cloud data, this embodiment needs to adjust the reconstructed infrared point cloud data to obtain initial infrared point cloud data. The initial infrared pixel coordinates corresponding to each pixel in the infrared image can also be determined, and an infrared mapping relationship can be constructed based on the initial infrared pixel coordinates and the initial infrared point cloud data. Finally, the corresponding infrared point cloud data set can be found from the infrared mapping relationship based on the set of infrared pixel coordinates corresponding to multiple infrared strings.

[0084] Further, in this embodiment, the step of adjusting the reconstructed infrared point cloud data to obtain initial infrared point cloud data specifically includes: determining the target Z-axis coordinates of all infrared pixels in the infrared image based on the initial Z-axis coordinates in the reconstructed infrared point cloud data; constructing a first relational expression based on the initial infrared pixel coordinates of all infrared pixels in the infrared image, the camera intrinsic parameter matrix, and the camera extrinsic parameter matrix; determining the camera projection matrix based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and converting the first relational expression into a second relational expression based on the camera projection matrix; and determining the initial infrared point cloud data corresponding to all infrared pixels in the infrared image based on the target Z-axis coordinates and the second relational expression.

[0085] Understandably, since the elevation difference within the pixel range of the photovoltaic string surface image is extremely small, this embodiment can obtain the initial Z-axis coordinate of the three-dimensional point cloud of the nearest pixel of any infrared pixel through the nearest neighbor search algorithm, and use the initial Z-axis coordinate as the target Z-axis coordinate of the infrared pixel.

[0086] It should be understood that the first relation is:

[0087]

[0088] Where s are the equation coefficients, the initial infrared pixel coordinates can be expressed as (u , v), the camera intrinsic parameter matrix recovered by SFM is Where c x c y f represents the coordinates of the infrared image center. x and f y The camera focal length is (X, Y, Z), and the camera extrinsic matrix may include the rotation matrix R and the translation vector T. (X, Y, Z) represents the initial infrared point cloud data.

[0089] In this embodiment, the camera projection matrix can be calculated based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix. Assume the projection matrix is... The first relation can be converted into the second relation, that is:

[0090]

[0091] Furthermore, the second relation above can be transformed into a third relation: (u*c 31 -c 11 )X+(u*c 32 -c 12 )Y+(u*c 33 -c 13 Z = (c 14 -u*c 34 ), and the fourth relation (v*c 31 -c 21 )X+(v*c 32 -c 22 )Y+(v*c 33 -c 23 Z = (c 24 -v*c 34 Since the fifth relation, a*X+b*Y+c*Z=1, can be combined with the third, fourth, and fifth relations into a matrix equation to obtain the sixth relation, i.e.:

[0092]

[0093] Where Z is the Z-axis coordinate of the target, X and Y can be obtained through the sixth relation, thus forming (X,Y,Z) corresponding to all infrared pixels, which is the initial infrared point cloud data.

[0094] Step S302: Determine the visible light point cloud data set based on the set of visible light pixel coordinates corresponding to the multiple visible light strings.

[0095] In a specific implementation, this embodiment can also perform SFM three-dimensional reconstruction on the visible light image to obtain the reconstructed visible light point cloud data corresponding to each pixel of the visible light image. Then, the reconstructed visible light point cloud data is adjusted using the method described above for adjusting the reconstructed infrared point cloud data to obtain initial visible light point cloud data. Furthermore, the initial visible light pixel coordinates corresponding to each pixel of the visible light image can be determined, and an infrared mapping relationship is constructed based on the initial visible light pixel coordinates and the initial visible light point cloud data. Finally, the corresponding visible light point cloud data set can be found from the visible light mapping relationship based on the set of visible light pixel coordinates corresponding to multiple visible light strings.

[0096] Step S303: Register the infrared point cloud data set and the visible light point cloud data set to obtain a registration relationship set.

[0097] Furthermore, in order to effectively register infrared point cloud data and visible light point cloud data, in this embodiment, step S303 includes: selecting a target infrared string from the plurality of infrared strings and selecting a target visible light string from the plurality of visible light strings; selecting target infrared point cloud data corresponding to the target infrared string from the infrared point cloud data set and selecting target visible light point cloud data corresponding to the target visible light string from the visible light point cloud data set; registering the target infrared point cloud data and the target visible light point cloud data to obtain a registration relationship; returning to the step of selecting the target infrared string from the plurality of infrared strings to obtain a new registration relationship and constructing a registration relationship set.

[0098] Understandably, a target infrared string can be arbitrarily selected from multiple infrared strings, and a target visible light string can be arbitrarily selected from multiple visible light strings. Furthermore, target infrared point cloud data corresponding to the target infrared string can be selected from an infrared point cloud dataset, and target visible light point cloud data corresponding to the target visible light string can be selected from a visible light point cloud dataset. (Refer to...) Figure 5 , Figure 5 This is a schematic diagram of target infrared point cloud data and target visible light point cloud data according to an embodiment of the image matching method of this application. Figure 5 The top left of the image shows the target infrared data string, the bottom left shows the corresponding target infrared point cloud data, the top right shows the target visible light data string, and the bottom right shows the corresponding target visible light point cloud data.

[0099] It should be understood that the target infrared point cloud data and the target visible light point cloud data can be registered using the Iterative Closest Point (ICP) registration algorithm to obtain the registration relationship, which can then be referenced. Figure 6 , Figure 6 This is a schematic diagram of point cloud data registration according to an embodiment of the image matching method of this application. Figure 6The following registration relationship is obtained after registering the target infrared point cloud data and the target visible light point cloud data.

[0100] In practice, after obtaining a registration relationship, the above steps can be repeated until all infrared and visible light strings are selected, resulting in several new registration relationships that constitute a set of registration relationships.

[0101] Furthermore, in order to register the target infrared point cloud data and the target visible light point cloud data, in this embodiment, the step of registering the target infrared point cloud data and the target visible light point cloud data to obtain a registration relationship specifically includes: registering the target infrared point cloud data and the target visible light point cloud data based on the iterative nearest point algorithm to obtain a registration relationship, wherein the registration relationship includes the mapping relationship between the target infrared point cloud data and the target visible light point cloud data.

[0102] Understandably, the target infrared point cloud data and the target visible light point cloud data can be registered based on the Iterative Closest Point (ICP) registration algorithm to obtain the registration relationship, which refers to the mapping relationship between the target point cloud data and the target visible light point cloud data.

[0103] This embodiment determines an infrared point cloud data set based on the set of infrared pixel coordinates corresponding to multiple infrared strings, and then determines a visible light point cloud data set based on the set of visible light pixel coordinates corresponding to multiple visible light strings. Finally, the infrared point cloud data set and the visible light point cloud data set are registered to obtain a registration relationship set. This embodiment first converts the set of infrared pixel coordinates corresponding to multiple infrared strings into an infrared point cloud data set, then converts the set of visible light pixel coordinates corresponding to multiple visible light strings into visible light point cloud data, and then registers the infrared point cloud data set corresponding to multiple infrared strings and the visible light point cloud data set corresponding to multiple visible light strings. This allows for point cloud registration using the infrared point cloud data set and the visible light point cloud data set.

[0104] refer to Figure 7 , Figure 7 This is a flowchart illustrating the third embodiment of the image matching method of this application.

[0105] Based on the above embodiments, in this embodiment, step S40 includes:

[0106] Step S401: Determine all registration relationships in the registration relationship set.

[0107] Step S402: Determine the registration infrared point cloud data and registration visible light point cloud data in each registration relationship.

[0108] It is understood that the registration relationship may include the mapping relationship between the target point cloud data and the target visible light point cloud data. Therefore, this embodiment can determine the registration infrared point cloud data and the registration visible light point cloud data in each registration relationship.

[0109] Step S403: Perform image matching on the infrared image and the visible light image based on the registered infrared point cloud data and the registered visible light point cloud data.

[0110] It should be understood that infrared and visible light images can be matched based on registered infrared point cloud data and registered visible light point cloud data. Specifically, the point cloud data can be converted into pixel coordinates for image matching.

[0111] Furthermore, in order to effectively perform image matching between the infrared image and the visible light image, in this embodiment, step S403 includes: determining the coordinates of the registered infrared pixel points based on the registered infrared point cloud data and the infrared mapping relationship; determining the coordinates of the registered visible light pixel points based on the registered visible light point cloud data and the visible light mapping relationship; and performing image matching between the infrared image and the visible light image based on the registered infrared pixel coordinates and the registered visible light pixel coordinates.

[0112] Understandably, the infrared mapping relationship is the mapping relationship between infrared point cloud data and infrared pixel coordinates. Therefore, the coordinates of the registered infrared pixels can be found from the infrared mapping relationship based on the registered infrared point cloud data. Similarly, the visible light mapping relationship is the mapping relationship between visible light point cloud data and visible light pixel coordinates. Therefore, the coordinates of the registered visible light pixels can be found from the visible light mapping relationship based on the registered visible light point cloud data. Then, image matching is performed on the infrared and visible light images based on the registered infrared and visible light pixel coordinates, referring to... Figure 8 , Figure 8 This is a schematic diagram illustrating the matching of infrared and visible light images according to an embodiment of the image matching method of this application.

[0113] This embodiment determines all registration relationships in the registration relationship set, then identifies the registered infrared point cloud data and registered visible light point cloud data within each registration relationship, and finally performs image matching between the infrared and visible light images based on the registered infrared and visible light point cloud data. This embodiment converts two-dimensional image matching into three-dimensional point cloud registration, achieving the matching of infrared and visible light images.

[0114] Reference Figure 9 , Figure 9 This is a structural block diagram of the first embodiment of the image matching device of this application.

[0115] like Figure 9 As shown, the image matching device proposed in this application includes:

[0116] Image acquisition module 10 is used to acquire infrared and visible light images of the photovoltaic inspection area;

[0117] The string segmentation module 20 is used to perform string segmentation on the infrared image and the visible light image respectively, to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image;

[0118] Data registration module 30 is used to register the infrared point cloud data set corresponding to the multiple infrared strings and the visible light point cloud data set corresponding to the multiple visible light strings to obtain a registration relationship set.

[0119] The image matching module 40 is used to perform image matching on the infrared image and the visible light image according to the registration relationship set.

[0120] This embodiment acquires infrared and visible light images of a photovoltaic inspection area, then performs string segmentation on both infrared and visible light images to obtain multiple infrared strings and visible light strings respectively. Next, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings to obtain a registration relationship set. Finally, it performs image matching on the infrared and visible light images based on the registration relationship set. This embodiment first performs string segmentation on the infrared and visible light images, enabling pixel semantic segmentation of both images to obtain multiple infrared and visible light strings. Then, it registers the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings, enabling point cloud registration using these data sets. Finally, it achieves image matching between the infrared and visible light images based on the registration relationship set.

[0121] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0122] In addition, for technical details not described in detail in this embodiment, please refer to the image matching method provided in any embodiment of this application, which will not be repeated here.

[0123] Based on the first embodiment of the image matching device described in this application, a second embodiment of the image matching device of this application is proposed.

[0124] In this embodiment, the data registration module 30 is further configured to determine an infrared point cloud data set based on the set of infrared pixel coordinates corresponding to the plurality of infrared strings; determine a visible light point cloud data set based on the set of visible light pixel coordinates corresponding to the plurality of visible light strings; and register the infrared point cloud data set and the visible light point cloud data set to obtain a registration relationship set.

[0125] Furthermore, the data registration module 30 is also used to perform three-dimensional reconstruction of the infrared image to obtain reconstructed infrared point cloud data; adjust the reconstructed infrared point cloud data to obtain initial infrared point cloud data; construct an infrared mapping relationship between the initial infrared pixel coordinates of all infrared pixels in the infrared image and the initial infrared point cloud data; and determine the infrared point cloud data set according to the set of infrared pixel coordinates corresponding to the multiple infrared strings and the infrared mapping relationship.

[0126] Furthermore, the data registration module 30 is also used to determine the target Z-axis coordinates of all infrared pixels in the infrared image based on the initial Z-axis coordinates in the reconstructed infrared point cloud data; construct a first relation based on the initial infrared pixel coordinates of all infrared pixels in the infrared image, the camera intrinsic parameter matrix, and the camera extrinsic parameter matrix; determine the camera projection matrix based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and convert the first relation into a second relation based on the camera projection matrix; and determine the initial infrared point cloud data corresponding to all infrared pixels in the infrared image based on the target Z-axis coordinates and the second relation.

[0127] Furthermore, the data registration module 30 is also used to select a target infrared string from the plurality of infrared strings and a target visible light string from the plurality of visible light strings; select target infrared point cloud data corresponding to the target infrared string from the infrared point cloud data set and select target visible light point cloud data corresponding to the target visible light string from the visible light point cloud data set; register the target infrared point cloud data and the target visible light point cloud data to obtain a registration relationship; return to the step of selecting the target infrared string from the plurality of infrared strings to obtain a new registration relationship and construct a registration relationship set.

[0128] Furthermore, the data registration module 30 is also used to register the target infrared point cloud data and the target visible light point cloud data based on the iterative nearest point algorithm to obtain a registration relationship, wherein the registration relationship includes the mapping relationship between the target infrared point cloud data and the target visible light point cloud data.

[0129] Furthermore, the image matching module 40 is also used to determine all registration relationships in the registration relationship set; determine the registration infrared point cloud data and registration visible light point cloud data in each registration relationship; and perform image matching on the infrared image and the visible light image based on the registration infrared point cloud data and the registration visible light point cloud data.

[0130] Furthermore, the image matching module 40 is also used to determine the coordinates of the registered infrared pixel points based on the registered infrared point cloud data and the infrared mapping relationship; determine the coordinates of the registered visible light pixel points based on the registered visible light point cloud data and the visible light mapping relationship; and perform image matching on the infrared image and the visible light image based on the registered infrared pixel point coordinates and the registered visible light pixel point coordinates.

[0131] Other embodiments or specific implementations of the image matching device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0132] This application provides an image matching device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image matching method in Embodiment 1 above.

[0133] The following is for reference. Figure 10 The diagram illustrates a structural schematic of an image matching device suitable for implementing embodiments of this application. The image matching device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The image matching device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0134] like Figure 10As shown, the image matching device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the image matching device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the image matching device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows image matching devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0135] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0136] The image matching device provided in this application, employing the image matching method described in the above embodiments, can solve the technical problem of how to effectively match infrared images and visible light images. Compared with the prior art, the beneficial effects of the image matching device provided in this application are the same as those of the image matching method described in the above embodiments, and other technical features of this image matching device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0137] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0138] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0139] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image matching method in the above embodiments.

[0140] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0141] The aforementioned computer-readable storage medium may be included in the image matching device; or it may exist independently and not be assembled into the image matching device.

[0142] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the image matching device, the image matching device causes the following actions: to acquire infrared images and visible light images of the photovoltaic inspection area; to perform string segmentation on the infrared images and the visible light images respectively, obtaining multiple infrared strings in the infrared images and multiple visible light strings in the visible light images; to register the infrared point cloud data sets corresponding to the multiple infrared strings and the visible light point cloud data sets corresponding to the multiple visible light strings, obtaining a registration relationship set; and to perform image matching on the infrared images and the visible light images according to the registration relationship set.

[0143] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0145] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0146] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described image matching method, thereby solving the technical problem of how to effectively match infrared images and visible light images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the image matching method provided in the above embodiments, and will not be repeated here.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image matching method described above.

[0148] The computer program product provided in this application can solve the technical problem of how to effectively match infrared images and visible light images. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the image matching method provided in the above embodiments, and will not be repeated here.

[0149] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image matching method characterized by, The image matching method includes the following steps: Collect infrared and visible light images of the photovoltaic inspection area; The infrared image and the visible light image are respectively segmented into multiple infrared strings in the infrared image and multiple visible light strings in the visible light image; Register the infrared point cloud data set corresponding to the plurality of infrared strings and the visible light point cloud data set corresponding to the plurality of visible light strings to obtain a registration relationship set; Image matching is performed on the infrared image and the visible light image based on the registration relationship set.

2. The image matching method of claim 1, wherein, The step of registering the infrared point cloud data sets corresponding to the plurality of infrared strings and the visible light point cloud data sets corresponding to the plurality of visible light strings to obtain a registration relationship set specifically includes: The infrared point cloud data set is determined based on the set of infrared pixel coordinates corresponding to the multiple infrared strings; The visible light point cloud data set is determined based on the set of visible light pixel coordinates corresponding to the multiple visible light strings; The infrared point cloud data set and the visible light point cloud data set are registered to obtain a set of registration relationships.

3. The image matching method of claim 2, wherein, The step of determining the infrared point cloud data set based on the set of infrared pixel coordinates corresponding to the multiple infrared strings specifically includes: The infrared image is reconstructed in three dimensions to obtain the reconstructed infrared point cloud data; The reconstructed infrared point cloud data is adjusted to obtain initial infrared point cloud data; Construct an infrared mapping relationship between the initial infrared pixel coordinates of all infrared pixels in the infrared image and the initial infrared point cloud data; The infrared point cloud data set is determined based on the set of infrared pixel coordinates corresponding to the multiple infrared strings and the infrared mapping relationship.

4. The image matching method of claim 3, wherein, The step of adjusting the reconstructed infrared point cloud data to obtain initial infrared point cloud data specifically includes: The target Z-axis coordinates of all infrared pixels in the infrared image are determined based on the initial Z-axis coordinates in the reconstructed infrared point cloud data. A first relation is constructed based on the initial infrared pixel coordinates of all infrared pixels in the infrared image, the camera intrinsic parameter matrix, and the camera extrinsic parameter matrix. The camera projection matrix is ​​determined based on the camera intrinsic parameter matrix and the camera extrinsic parameter matrix, and the first relation is converted into a second relation based on the camera projection matrix; The initial infrared point cloud data corresponding to all infrared pixels in the infrared image is determined based on the target Z-axis coordinates and the second relational formula.

5. The image matching method of claim 2, wherein, The step of registering the infrared point cloud data set and the visible light point cloud data set to obtain a registration relationship set specifically includes: Select a target infrared string from the plurality of infrared strings, and select a target visible light string from the plurality of visible light strings; Select the target infrared point cloud data corresponding to the target infrared string from the infrared point cloud data set, and select the target visible light point cloud data corresponding to the target visible light string from the visible light point cloud data set; The infrared point cloud data and the visible light point cloud data of the target are registered to obtain the registration relationship; Return to the step of selecting the target infrared string from the plurality of infrared strings, obtain a new registration relationship, and construct a registration relationship set.

6. The image matching method of claim 5, wherein, The step of registering the target infrared point cloud data and the target visible light point cloud data to obtain the registration relationship specifically includes: The target infrared point cloud data and the target visible light point cloud data are registered based on the iterative nearest point algorithm to obtain the registration relationship, which includes the mapping relationship between the target infrared point cloud data and the target visible light point cloud data.

7. The image matching method of claim 4, wherein, The step of performing image matching between the infrared image and the visible light image based on the registration relationship set specifically includes: Determine all registration relationships in the registration relationship set; Determine the registration infrared point cloud data and registration visible light point cloud data in each registration relationship; Image matching is performed on the infrared image and the visible light image based on the registered infrared point cloud data and the registered visible light point cloud data.

8. The image matching method of claim 7, wherein, The step of performing image matching on the infrared image and the visible light image based on the registered infrared point cloud data and the registered visible light point cloud data specifically includes: The coordinates of the registered infrared pixel points are determined based on the registered infrared point cloud data and the infrared mapping relationship. The coordinates of the registered visible light pixels are determined based on the registered visible light point cloud data and the visible light mapping relationship. Image matching is performed on the infrared image and the visible light image based on the registered infrared pixel coordinates and the registered visible light pixel coordinates.

9. An image matching apparatus characterized by comprising: The image matching device includes: The image acquisition module is used to acquire infrared and visible light images of the photovoltaic inspection area; The string segmentation module is used to perform string segmentation on the infrared image and the visible light image respectively, to obtain multiple infrared strings in the infrared image and multiple visible light strings in the visible light image; The data registration module is used to register the infrared point cloud data set corresponding to the multiple infrared strings and the visible light point cloud data set corresponding to the multiple visible light strings to obtain a registration relationship set. An image matching module is used to perform image matching between the infrared image and the visible light image based on the registration relationship set.

10. An image matching device, characterized by The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image matching method as described in any one of claims 1 to 8.

11. A storage medium, characterized by The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image matching method as described in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the image matching method as described in any one of claims 1 to 8.

Images