Image Processing Method, Apparatus, Storage Medium, Program Product, and Electronic Device

Abstract

The present application discloses an image processing method, apparatus, storage medium, program product and electronic device. The method includes: determining matching feature point pairs between a first orthographic image and each second orthographic image; respectively determining first mapping relationships on the images between the first orthographic image and each second orthographic image based on the matching feature point pairs between the first orthographic image and each second orthographic image; determining the pixel coordinates of corresponding points in the first orthographic image based on the first mapping relationships corresponding to each second orthographic image and the pixel coordinates of the image center points of each second orthographic image; determining a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image based on the pixel coordinates of the corresponding points and the geographic coordinates of the image center points; and determining the geographic range of the first orthographic image based on the pixel coordinates of the positioning points of the first orthographic image and the second mapping relationship, so as to accurately determine the geographic range of the first orthographic image.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and in particular, relates to an image processing method, apparatus, storage medium, program product, and electronic device. Background Art

[0002] With the wide promotion of new energy technologies, photovoltaic power generation, as a representative of clean energy, has enabled the construction of photovoltaic power stations to be carried out at a rapid pace. Currently, the daily management and maintenance of photovoltaic power stations are usually based on the images of the photovoltaic power stations taken.

[0003] However, currently, it is impossible to quickly and accurately estimate the actual geographical range corresponding to the images of the photovoltaic power stations taken, which directly affects key links such as subsequent image stitching, 3D modeling, and precise sorting of image information, causing certain difficulties in the daily management and maintenance of photovoltaic power stations. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, storage medium, program product, and electronic device, which can quickly and accurately estimate the geographical range corresponding to an orthographic image.

[0005] In a first aspect, embodiments of this application provide an image processing method, including: obtaining a first orthographic image and multiple second orthographic images, where there are image overlapping regions between the first orthographic image and each second orthographic image; performing image matching on the first orthographic image and each second orthographic image to determine matching feature point pairs between the first orthographic image and each second orthographic image; based on the matching feature point pairs between the first orthographic image and each second orthographic image, respectively determining first mapping relationships on the image between the first orthographic image and each second orthographic image; based on the first mapping relationships corresponding to each second orthographic image and the pixel coordinates of the image center points of each second orthographic image, determining the pixel coordinates of corresponding points in the first orthographic image, where each corresponding point corresponds one-to-one to the image center point of each second orthographic image; based on the pixel coordinates of each corresponding point and the geographical coordinates of each image center point, determining a second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image; based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship, determining the geographical range of the first orthographic image.

[0006] In an optional implementation manner of the first aspect, respectively determining the first mapping relationships on the image between the first orthographic image and each second orthographic image includes: constructing a first transformation matrix, where the first transformation matrix contains multiple transformation parameters; for the first orthographic image and any one of the second orthographic images, based on the first transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any one of the second orthographic images, determining the parameter values of the multiple transformation parameters of the first transformation matrix, and the first transformation matrix is used to represent the first mapping relationship.

[0007] In an alternative embodiment of the first aspect, the process of determining the geographical coordinates of the image center points of the second orthographic images includes: determining the geographical coordinates of the second orthographic images based on the longitude and latitude parameters of the second orthographic images; and determining the geographical coordinates of the image center points of the second orthographic images as the geographical coordinates of the second orthographic images.

[0008] In an alternative embodiment of the first aspect, determining the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image based on the pixel coordinates of the corresponding points and the geographical coordinates of the image center points includes: using the geographical coordinates of the image center points as the geographical coordinates of the corresponding points; constructing a second transformation matrix, where the second transformation matrix contains multiple transformation parameters; and determining the parameter values of the multiple transformation parameters of the second transformation matrix based on the second transformation matrix, the geographical coordinates, and the pixel coordinates of the corresponding points, where the second transformation matrix is used to represent the second mapping relationship.

[0009] In an alternative embodiment of the first aspect, the positioning points are at least three pixel points that are not on the same straight line.

[0010] In an alternative embodiment of the first aspect, when the positioning point is a corner point of the first orthographic image, determining the geographical range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship includes: obtaining the geographical coordinates of the corner points of the first orthographic image based on the pixel coordinates of the corner points of the first orthographic image and the second mapping relationship; and determining the geographical range of the first orthographic image based on the geographical coordinates of the corner points of the first orthographic image.

[0011] In an alternative embodiment of the first aspect, obtaining the first orthographic image and multiple second orthographic images includes: screening out multiple second orthographic images that have an image overlap region with the first orthographic image from multiple orthographic images.

[0012] In an alternative embodiment of the first aspect, screening out multiple second orthographic images that have an image overlap region with the first orthographic image from multiple orthographic images includes: determining the geographical distance between the first orthographic image and each orthographic image based on the geographical coordinates of the first orthographic image and each orthographic image; determining whether the geographical distance meets a preset condition; and when the geographical distance meets the preset condition, determining the orthographic image as a second orthographic image.

[0013] In an alternative embodiment of the first aspect, the image processing method further includes: performing distortion correction on the first orthographic image and multiple second orthographic images to obtain the distortion-corrected first orthographic image and multiple second orthographic images, where the distortion-corrected first orthographic image and multiple second orthographic images are used to extract feature points.

[0014] In a second aspect, an embodiment of the present application provides an image processing apparatus, including: an acquisition unit configured to acquire a first orthographic image and a plurality of second orthographic images, where there is an image overlapping area between the first orthographic image and each second orthographic image; a matching unit configured to perform image matching on the first orthographic image and each second orthographic image to determine pairs of feature points that match between the first orthographic image and each second orthographic image; a first determination unit configured to respectively determine a first mapping relationship on the image between the first orthographic image and each second orthographic image based on the pairs of feature points that match between the first orthographic image and each second orthographic image; a second determination unit configured to determine the pixel coordinates of corresponding points in the first orthographic image based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center points of each second orthographic image, where each corresponding point corresponds one-to-one to the image center point of each second orthographic image; a third determination unit configured to determine a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point; a fourth determination unit configured to determine the geographic range of the first orthographic image based on the pixel coordinates of the positioning points of the first orthographic image and the second mapping relationship.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the image processing method of the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the image processing method of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides an electronic device. The electronic device includes a processor and a memory storing computer program instructions, and the processor is coupled to the memory; when the processor executes the computer program instructions, the image processing method as in the first aspect is implemented.

[0018] In the embodiments of the present application, since there are image overlapping regions between the first orthographic image and each second orthographic image, image matching is performed on the first orthographic image and multiple second orthographic images, so that feature point pairs with a mapping relationship between the first orthographic image and each second orthographic image can be determined. By using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, that is, the first mapping relationship, can be accurately and efficiently determined. Thus, by using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped to the first orthographic image, and in the first orthographic image, a corresponding point corresponding to the image center point can be determined. Since the geographical coordinates of the image center points of each second orthographic image are known accurately, and the image center points of each second orthographic image and the corresponding points in the first orthographic image are in one-to-one correspondence, the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image can be accurately determined based on the pixel coordinates of each corresponding point and the geographical coordinates of each image center point. Since the second mapping relationship accurately describes the mapping relationship between the image space and the physical space of the first orthographic image, thus, based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship, the geographical range of the first orthographic image can be quickly and accurately determined. It can be seen that in the process of determining the geographical range of the first orthographic image in the present application, the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the image center points are cleverly utilized to quickly and accurately determine the geographical range of the first orthographic image. On this basis, the multiple images of the photovoltaic power station taken can be processed such as mosaicked by using the accurately estimated geographical range, and the overall area of the photovoltaic power station can be analyzed and the problem location can be carried out, effectively improving the daily management and maintenance efficiency of the photovoltaic power station. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

[0020] Figure 1 is a schematic diagram of the architecture of an image processing system provided by an embodiment of the present application;

[0021] Figure 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0022] Figure 3 is a flowchart of determining the first mapping relationship based on perspective transformation provided by an embodiment of the present application;

[0023] Figure 4It is a flowchart for determining a second mapping relationship based on perspective transformation provided by an embodiment of the present application;

[0024] Figure 5 It is a flowchart for determining the geographical range of a first orthographic image provided by an embodiment of the present application;

[0025] Figure 6 It is a flowchart for an image processing method in the scenario where a drone inspects a photovoltaic power station provided by an embodiment of the present application;

[0026] Figure 7 It is a schematic structural diagram of an image processing device provided by another embodiment of the present application;

[0027] Figure 8 It is a schematic structural diagram of an electronic device provided by yet another embodiment of the present application.

[0028] Among them, the above-mentioned drawings include the following reference numerals:

[0029] 100, electronic device; 200, drone; 700, image processing device; 710, acquisition unit; 720, matching unit; 730, first determination unit; 740, second determination unit; 750, third determination unit; 760, fourth determination unit; 801, processor; 802, memory; 803, communication interface; 810, bus. Detailed Description of the Invention

[0030] The features and exemplary embodiments of various aspects of the present application will be described in detail below. For the purpose of making the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only intended to provide a better understanding of the present application by showing examples of the present application.

[0031] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0032] With the continuous expansion of renewable energy technology, photovoltaic power stations are being comprehensively constructed. Usually, based on the images of photovoltaic power stations taken, the daily management and maintenance of photovoltaic power stations are carried out. However, currently, it is impossible to quickly and accurately estimate the actual geographical scope corresponding to the images of photovoltaic power stations taken, resulting in the inability to splice, three-dimensionally model, and precisely sort out image information for multiple images of photovoltaic power stations taken using the accurately estimated geographical scope. This causes certain difficulties in analyzing the overall area of photovoltaic power stations and locating problems, thereby leading to relatively low efficiency in the daily management and maintenance of photovoltaic power stations. For example, in the scenario where a drone conducts inspections on a photovoltaic power station, the image acquisition device configured on the drone can take images of the photovoltaic power station along the inspection route. However, due to the inability to quickly and accurately estimate the geographical scope corresponding to each image, it is impossible to use the accurately estimated geographical scope to splice the images of the photovoltaic power station taken by the drone, thus making it impossible to precisely analyze the overall area of the photovoltaic power station and locate problems.

[0033] The present application provides an image processing method, apparatus, storage medium, program product, and electronic device. Since a first orthographic image and each second orthographic image have an image overlap area, image matching is performed on the first orthographic image and multiple second orthographic images to determine feature point pairs with a mapping relationship between the first orthographic image and each second orthographic image. Using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, i.e., the first mapping relationship, can be accurately and efficiently determined. Thus, using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped to the first orthographic image, and the corresponding point corresponding to the image center point is determined in the first orthographic image. Since the geographic coordinates of the image center point of each second orthographic image are known accurately, and the image center point of each second orthographic image corresponds one-to-one with the corresponding point in the first orthographic image, the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image can be accurately determined based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point. Since the second mapping relationship accurately describes the mapping relationship between the image space of the first orthographic image and the physical space, the geographical scope of the first orthographic image can be quickly and accurately determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship. It can be seen that in the process of determining the geographical scope of the first orthographic image, the present application cleverly utilizes the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the image center point to quickly and accurately determine the geographical scope of the first orthographic image. On this basis, the accurately estimated geographical scope can be used to stitch and process multiple images of the photovoltaic power station, analyze the overall area of the photovoltaic power station and locate problems, and effectively improve the daily management and maintenance efficiency of the photovoltaic power station.

[0034] For ease of understanding, here we take the example of drone inspection of photovoltaic power stations to briefly introduce the application architecture of the image processing method, device, storage medium, program product and electronic device provided in this application. Figure 1 A schematic diagram of the application architecture of an image processing system provided in an embodiment of the present application. The image processing system includes a drone 200 and an electronic device 100. The drone 200 inspects the photovoltaic power station based on the inspection route and uses the image acquisition device configured for the drone to capture multiple images. It should be noted that when capturing multiple images, the image acquisition device of the drone keeps the lens pointing vertically downward at the ground to ensure that the multiple images captured are orthographic images. After receiving the multiple orthographic images captured by the drone 200, the electronic device 100 processes the multiple orthographic images according to the image processing method provided in the present application to determine the geographical range corresponding to each orthographic image.

[0035] In practical applications, in the embodiments of the present application, the image processing method provided by the embodiments of the present application can be implemented when a processor of an electronic device executes a program or an instruction. However, in some embodiments, an image acquisition device of a drone or other devices can also have similar functions. For example, the image processing method provided by the embodiments of the present application can be implemented when an image acquisition device of a drone executes a program or an instruction. The embodiments of the present application do not limit this.

[0036] It should be noted that the application scenarios described in the above embodiments of the present application are for more clearly explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems. The image processing method provided by the embodiments of the present application can be applied to various application scenarios that require accurate estimation of the geographical range of an image.

[0037] The image processing method, device, storage medium, program product, and electronic device provided by the present application will be described separately below.

[0038] Figure 2 It is a schematic flowchart of an image processing method provided by an embodiment of the present application. As Figure 2 shown, the image processing method provided by the embodiments of the present application may include steps S201 to S206.

[0039] In step S201, a first orthographic image and multiple second orthographic images are acquired, and there is an image overlap area between the first orthographic image and each second orthographic image.

[0040] Among them, the first orthographic image and the multiple second orthographic images are taken when the lens of the image acquisition device of the drone is vertically downward and aligned with the ground.

[0041] The image overlap area refers to the area with the same or similar content in the first orthographic image and each second orthographic image. Optionally, the image overlap between the first orthographic image and each second orthographic image can be complete overlap or partial overlap.

[0042] In some embodiments, any one of the multiple orthographic images can be determined as the first orthographic image, and then, from the orthographic images other than the first orthographic image, multiple orthographic images having an image overlap area with the first orthographic image are selected as the multiple second orthographic images.

[0043] Taking the inspection of a photovoltaic power station by a drone as an example, in practical applications, when the drone executes an inspection task, the image acquisition device configured on the drone takes pictures of the photovoltaic power station, obtaining an orthographic image set containing n orthographic images 。For example, when estimating the geographical range, can be used as the first orthophoto image, and multiple second orthophoto images with image overlap regions with can be determined from { }. When estimating the geographical range, can be used as the first orthophoto image, and multiple second orthophoto images with image overlap regions with can be determined from }.

[0044] Since there are image overlap regions between the first orthophoto image and multiple second orthophoto images, this ensures that there are many common features between the first orthophoto image and each second orthophoto image. Therefore, feature point pairs can be accurately determined between the first orthophoto image and each second orthophoto image subsequently.

[0045] There can be multiple implementation methods for screening multiple second orthophoto images with image overlap regions with the first orthophoto image from multiple orthophoto images.

[0046] In one embodiment, a neural network model can be used to perform object detection on the first orthophoto image and multiple orthophoto images, and then determine the intersection over union (IOU) values of the bounding boxes obtained by object detection in the first orthophoto image and the bounding boxes obtained by object detection in each orthophoto image. Finally, based on the IOU values between the first orthophoto image and each orthophoto image, the orthophoto images corresponding to the IOU values greater than a preset threshold are determined as the second orthophoto images.

[0047] In another embodiment, the second orthophoto images with image overlap regions with the first orthophoto image can be screened based on the continuous shooting time screening method. For example, the orthophoto images with a shooting time interval less than a preset threshold from the first orthophoto image are determined as the second orthophoto images.

[0048] In yet another embodiment, the second orthophoto images with image overlap regions with the first orthophoto image can be determined based on the geographical distance between the first orthophoto image and each orthophoto image. Specifically, based on the geographical coordinates of the first orthophoto image and each orthophoto image, the geographical distance between the first orthophoto image and each orthophoto image is determined; it is determined whether the geographical distance meets the preset conditions; and when the geographical distance meets the preset conditions, the orthophoto image is determined as the second orthophoto image.

[0049] Since the geographical coordinates of the first orthophoto image and each orthophoto image are two-dimensional plane coordinates, , determine the geographical distance between the first orthophoto image and each orthophoto image, where ( , ) is used to represent the geographical coordinates of the first orthophoto image, is used to represent the abscissa of the first orthophoto image in the geographical coordinate system, is used to represent the ordinate of the first orthophoto image in the geographical coordinate system, ( , ) is used to represent the geographical coordinates of the orthophoto image, is used to represent the abscissa of the orthophoto image in the geographical coordinate system, is used to represent the ordinate of the orthophoto image in the geographical coordinate system.

[0050] The preset condition can be that the geographical distance between the first orthophoto image and each orthophoto image is less than the distance threshold. The setting of the distance threshold can be determined based on the shooting height of the image acquisition device, the shooting angle of the image acquisition device, the focal length of the image acquisition device, etc.

[0051] Based on the geographical distance between the first orthophoto image and each orthophoto image, screen each orthophoto image, so that the second orthophoto image with an image overlap area with the first orthophoto image can be quickly determined.

[0052] In step S202, perform image matching on the first orthophoto image and each second orthophoto image to determine the corresponding feature point pairs between the first orthophoto image and each second orthophoto image.

[0053] Among them, image matching is to determine the feature points of the similar parts between different images and establish the corresponding relationship of the feature points of the similar parts between different images. Usually, image matching of two images can be performed based on methods such as feature point matching, region matching, or global matching.

[0054] In one embodiment, taking the inspection of a photovoltaic power station by a drone as an example, image matching is further described. First, taking a first orthophoto image and a second orthophoto image as examples, extract the feature points in the first orthophoto image and the feature points in the second orthophoto image. Then, based on the feature vector distance between the feature points in the first orthophoto image and the feature points in the second orthophoto image, determine whether the feature points in the first orthophoto image and the feature points in the second orthophoto image correspond to the same feature. If the feature vector distance between the feature points in the first orthophoto image and the feature points in the second orthophoto image meets the requirements, that is, the two feature points correspond to the same feature, then there is a corresponding relationship between the feature points in the first orthophoto image and the feature points in the second orthophoto image, and thus the feature points in the first orthophoto image and the feature points in the second orthophoto image are determined as feature point pairs.

[0055] A feature point pair refers to two feature points with a corresponding relationship in two images.

[0056] In the actual application process, any feasible feature extraction method in the prior art can be used to extract the feature points in the first orthographic image or the second orthographic image. For example, algorithms such as the Scale Invariant Feature Transform (SIFT), the Oriented FAST and Rotated BRIEF (ORB), or the Binary Robust Invariant Scalable Keypoints (BRISK) can be used to extract the feature points in the first orthographic image or the second orthographic image.

[0057] In an alternative embodiment, taking the SIFT algorithm as an example, the process of image matching between the first orthographic image and each second orthographic image to determine the feature point pairs between the first orthographic image and each second orthographic image will be further described. This process includes a feature extraction step, a feature matching step, a feature matching result screening step, and a false match removal step.

[0058] Feature extraction step: Extract the SIFT feature points of the first orthographic image and each second orthographic image to obtain a feature point set SIFT = { , , , …, }. Each element in the feature point set SIFT is used to represent the SIFT feature points in the first orthographic image or the second orthographic image, and it is, for example but not limited to, in the form of a feature vector.

[0059] Feature matching step: Using the nearest neighbor matching method, determine the two feature points with the closest feature vector distance to the feature points in the first orthographic image from each second orthographic image.

[0060] Feature matching result screening step: For the sake of easy understanding, one feature point in the first orthographic image and the two feature points with the closest feature vector distance in a corresponding second orthographic image will be described. Using the set feature threshold , screen the feature vector distance between one feature point in the first orthographic image and the two feature points in a corresponding second orthographic image, and filter out the feature vector distances greater than the feature threshold Feature points. At the same time, determine the ratio of the feature vector distance between a feature point of the first orthographic image and the nearest feature point in the second orthographic image to the feature vector distance between a feature point of the first orthographic image and the second-nearest feature point in the second orthographic image. When this ratio is less than the set ratio threshold in the case of, determine a feature point in the first orthographic image and the nearest feature point in the second orthographic image as a feature point pair. For example, this ratio threshold can be set to 0.75.

[0061] Mismatch removal step: Use the Random Sample Consensus (RANSAC) algorithm to remove mismatched feature point pairs. RANSAC estimates a geometric transformation model (such as a homography matrix) by iteratively selecting feature point pairs, and then validates other feature point pairs according to this geometric transformation model, excluding feature point pairs that do not conform to the geometric transformation model.

[0062] Based on the above process, feature point pairs between the first orthographic image and each second orthographic image can be obtained. Use to represent the first orthographic image and the second orthographic image matched feature point pairs between, the pixel coordinates of the feature points in the first orthographic image can be expressed as:

[0063]

[0064] The pixel coordinates of the feature points in the second orthographic image corresponding one-to-one to the feature points in the first orthographic image can be expressed as:

[0065]

[0066] Among them, is used to represent the pixel coordinates of feature points in the first orthographic image , ([[]] , ) is used to represent the pixel coordinates of the th feature point in the first orthographic image is used to represent the th feature point in the first orthographic image is used to represent the th feature point in the image coordinate system of the first orthographic image For representing the pixel coordinates of the feature points in the second orthographic image in which, ( , ) is used to represent the pixel coordinates of the th feature point in the second orthographic image, is used to represent the abscissa of the th feature point under the image coordinates of the second orthographic image, is used to represent the ordinate of the th feature point under the image coordinates of the second orthographic image.

[0067] In step S203, based on the pairs of matching feature points between the first orthographic image and each second orthographic image, the first mapping relationship on the image between the first orthographic image and each second orthographic image is determined respectively.

[0068] Among them, the first mapping relationship can have the following forms: transformation matrix, homography matrix, function, lookup table, point-to-point matching list, and so on.

[0069] In one embodiment, based on the pixel coordinates of the pairs of matching feature points between the first orthographic image and each second orthographic image, the first mapping relationship on the image between the first orthographic image and each second orthographic image can be determined. In this solution, the pixel coordinates of the pairs of feature points between the first orthographic image and each second orthographic image are accurate. Therefore, the mapping relationship between the first orthographic image and each second orthographic image, that is, the first mapping relationship, can be determined accurately and efficiently.

[0070] In step S204, based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center points of each second orthographic image, the pixel coordinates of the corresponding points in the first orthographic image are determined, and the corresponding points correspond one-to-one with the image center points of each second orthographic image.

[0071] Among them, the image center point of the second orthographic image is the point at the exact center of the second orthographic image. This image center point can be a feature point. Of course, this image center point can also be a pixel point other than the feature point.

[0072] The corresponding point in the first orthographic image can be a feature point. Of course, this corresponding point can also be a pixel point other than the feature point.

[0073] In one embodiment, let w represent the height of the orthographic image and h represent the width of the orthographic image. If the pixel coordinates of the image center point of the second orthographic image are represented in matrix form, the pixel coordinates of this image center point can be represented as Based on the first mapping relationship, the pixel coordinates of the center point of the image are mapped to the first orthographic image to obtain a corresponding point, which is represented as Thus, based on the first mapping relationship and the pixel coordinates of the center point of the second orthographic image, the calculation formula for determining the pixel coordinates of the corresponding point can be expressed as:

[0074] s =

[0075] where s = + + 1, is used to represent the first mapping relationship. In this example, can be in the form of a matrix, where and are the parameters of the last row of the matrix, and ([[]] , ) is used to represent the pixel coordinates of the corresponding point in the first orthographic image, is used to represent the abscissa of the corresponding point in the image coordinate system of the first orthographic image, is used to represent the ordinate of the corresponding point in the image coordinate system of the first orthographic image.

[0076] Since the first mapping relationship between the first orthographic image and each second orthographic image has been determined in step S203, therefore, based on the first mapping relationship corresponding to each second orthographic image, the center points of the images on each second orthographic image can be quickly mapped to the first orthographic image, and in the first orthographic image, the corresponding points corresponding to the center points of the images are determined.

[0077] In step S205, based on the pixel coordinates of each corresponding point and the geographical coordinates of each image center point, the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image is determined.

[0078] Among them, the second mapping relationship can have the following forms: perspective transformation matrix, homography matrix, function, look-up table, point-to-point matching list, and so on.

[0079] In the scenario where the drone inspects the photovoltaic power station, since each second orthographic image is taken when the lens of the image acquisition device carried by the drone is vertically downward and aimed at the ground, the geographical coordinates of the second orthographic image can be directly determined as the geographical coordinates of the center point of the second orthographic image.

[0080] In one embodiment, the process of determining the geographical coordinates of the image center points of the second orthophotos includes: determining the geographical coordinates of the second orthophotos based on the longitude and latitude parameters of the second orthophotos; and determining the geographical coordinates of the image center points of the second orthophotos as the geographical coordinates of the second orthophotos.

[0081] There are various ways to obtain the geographical coordinates of the second orthophotos.

[0082] In one embodiment, the geographical coordinates of the second orthophotos can be directly obtained through an image acquisition device.

[0083] In another embodiment, the geographical coordinates of the second orthophotos can be determined based on the longitude and latitude parameters of the second orthophotos.

[0084] There are various implementation ways to determine the geographical coordinates of the second orthophotos based on the longitude and latitude parameters of the second orthophotos.

[0085] In one embodiment, the longitude and latitude parameters of the second orthophotos are converted to the China Geodetic Coordinate System 2000 (CGCS2000) coordinate system to obtain the geographical coordinates of the second orthophotos.

[0086] In another embodiment, the longitude and latitude parameters of the second orthophotos are converted to the World Geodetic System (WGS84) coordinate system to obtain the geographical coordinates of the second orthophotos.

[0087] In yet another embodiment, the longitude and latitude parameters of the second orthophotos are converted to the Beidou coordinate system to obtain the geographical coordinates of the second orthophotos.

[0088] In the scenario where a drone conducts inspections on a photovoltaic power station, since the first orthophoto is also taken by the lens of the image acquisition device configured on the drone vertically downward towards the ground, the geographical coordinates of the first orthophoto can be determined as the geographical coordinates of the image center point of the first orthophoto. The determination method of the geographical coordinates of the first orthophoto can refer to the determination method of the geographical coordinates of the second orthophoto described above and will not be elaborated here.

[0089] In one embodiment, the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthophoto can be determined based on the pixel coordinates and geographical coordinates of the image center point of the first orthophoto, as well as the pixel coordinates and geographical coordinates of the corresponding points in the first orthophoto.

[0090] Based on the first orthophoto and the longitude and latitude parameters of the second orthophotos, the geographical coordinates of the first orthophoto and the second orthophotos are determined, which can improve the accuracy of the geographical coordinates of the first orthophoto and the second orthophotos, thereby improving the accuracy of the second mapping relationship.

[0091] There are various ways to obtain the longitude and latitude parameters of each second orthophoto image.

[0092] In one embodiment, the longitude and latitude parameters of each second orthophoto image can be determined by manual annotation.

[0093] In another embodiment, it can be directly obtained from the Exchangeable Image File Format (EXIF) of each second orthophoto image. The EXIF file refers to the metadata embedded in the image. This EXIF file records the shooting conditions of the image, the relevant settings of the image acquisition device, and the detailed information of the editing software. And the EXIF file can be applied to various types of image formats, such as JPEG format, PNG format, and TIFF format.

[0094] In the scenario where the drone inspects the photovoltaic power station, if the first orthophoto image and the second orthophoto images collected by the image acquisition device configured on the drone are in JPEG format, the corresponding longitude and latitude parameters of the first orthophoto image and each second orthophoto image can be directly obtained from the EXIF files corresponding to the first orthophoto image and each second orthophoto image.

[0095] In step S206, based on the pixel coordinates of the positioning points in the first orthophoto image and the second mapping relationship, the geographical range of the first orthophoto image is determined.

[0096] Among them, according to actual needs, the geographical range of the first orthophoto image can be the geographical range of the real environment corresponding to the first orthophoto image, or it can be the geographical range of a part of the real environment corresponding to the first orthophoto image.

[0097] The positioning points in the first orthophoto image can be feature points. Of course, the positioning points can also be pixel points other than feature points.

[0098] In the actual application process, the geographical range of the first orthophoto image can be determined based on the pixel coordinates of different numbers of positioning points and the second mapping relationship.

[0099] In one embodiment, when the number of positioning points is two, the two positioning points can be located on the same diagonal line of the first orthophoto image.

[0100] In another embodiment, the positioning points can be at least three pixel points not on the same straight line. Since three points not on the same straight line can determine a plane, therefore, based on the pixel coordinates of three pixel points not on the same straight line in the first orthophoto image, the geographical range of the first orthophoto image can be determined relatively quickly.

[0101] For example, when the number of positioning points is three, any two of the three positioning points may be located on the same side of the first orthographic image, and the remaining one positioning point may be located on the opposite side of the side where any two positioning points are located.

[0102] For example, when the number of positioning points is four, the four positioning points may be the four corner points of the first orthographic image.

[0103] In the embodiments of the present application, since there are image overlapping regions between the first orthographic image and each second orthographic image, therefore, by performing image matching on the first orthographic image and multiple second orthographic images, feature point pairs with a mapping relationship between the first orthographic image and each second orthographic image can be determined. By using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, that is, the first mapping relationship, can be accurately and efficiently determined. Thus, by using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped into the first orthographic image, and in the first orthographic image, a corresponding point corresponding to the image center point can be determined. Since the geographical coordinates of the image center points of each second orthographic image are accurately known, and the image center points of each second orthographic image and the corresponding points in the first orthographic image are in one-to-one correspondence, therefore, based on the pixel coordinates of each corresponding point and the geographical coordinates of each image center point, the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image can be accurately determined. Since the second mapping relationship accurately describes the mapping relationship between the image space and the physical space of the first orthographic image, thus, based on the pixel coordinates of the positioning points of the first orthographic image and the second mapping relationship, the geographical range of the first orthographic image can be quickly and accurately determined. It can be seen that in the process of determining the geographical range of the first orthographic image in the present application, the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the image center points are cleverly utilized to quickly and accurately determine the geographical range of the first orthographic image. On this basis, the multiple images of the photographed photovoltaic power station can be processed such as spliced by using the accurately predicted geographical range, and the overall area of the photovoltaic power station can be analyzed and problem located, effectively improving the daily management and maintenance efficiency of the photovoltaic power station.

[0104] Geometric transformation of an image is used to describe changes in the position, size, shape, etc. of the image, which may include rigid transformation, similarity transformation, affine transformation, perspective transformation, etc. Among them, rigid transformation may involve translation and rotation between images, similarity transformation involves scaling and shearing between images, perspective transformation is to project the image onto a new viewing plane, also called projection mapping. Perspective transformation can simultaneously handle rotation, translation, scaling, and shearing between two images, while affine transformation is a linear transformation and can be regarded as a special case of perspective transformation. Even if the two images are taken by an image acquisition device at different positions, the correct mapping relationship between the two images can be accurately found.

[0105] In an embodiment of the present application, a first transformation matrix may be used to represent the first mapping relationship, where the first transformation matrix includes multiple transformation parameters. The following embodiments take the most complex perspective transformation as an example to describe the determination of the first mapping relationship between a first orthographic image and a second orthographic image, that is, using a first perspective transformation matrix including multiple perspective transformation parameters as an example of the first transformation matrix. However, it can be understood that under the overall concept of the present application, the perspective transformation can be replaced with other geometric transformations according to needs.

[0106] As Figure 3 shown, another embodiment of the present application provides a flowchart for determining the first mapping relationship based on perspective transformation. Figure 3 Different from other embodiments, step S203 in other embodiments can be specifically refined into Figure 3 step S2031 and step S2032 in

[0107] In step S2031, a first perspective transformation matrix is constructed, and the first perspective transformation matrix contains multiple perspective transformation parameters.

[0108] Among them, the perspective transformation parameters have properties such as rotation, translation, scaling, shearing, and perspective transformation.

[0109] In one embodiment, the constructed first perspective transformation matrix H may be a 3 × 3 matrix. Each element in the first perspective transformation matrix H is a perspective transformation parameter. For example, the constructed first perspective transformation matrix H may contain 8 unknown perspective transformation parameters, so the first perspective transformation matrix H can be expressed as

[0110]

[0111] where, as a non-limiting example, a, b, d, and e are perspective transformation parameters for rotation, scaling, and shearing, c and f are perspective transformation parameters for translation, and g and h are perspective transformation parameters for perspective transformation.

[0112] In step S2032, for the first orthographic image and any second orthographic image, based on the first perspective transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any second orthographic image, determine the parameter values of multiple perspective transformation parameters of the first perspective transformation matrix, where the first perspective transformation matrix is used to represent the first mapping relationship.

[0113] Since the first perspective transformation matrix is used to represent the mapping relationship between the corresponding first orthographic image and the second orthographic image on the image, therefore, based on the constructed first perspective transformation matrix containing unknown perspective transformation parameters and the feature point pairs between the corresponding first orthographic image and the second orthographic image, the mapping relationship between the first orthographic image and the second orthographic image can be constructed.

[0114] For example, taking the first orthographic image and the second orthographic image as an example, further illustrate the process of determining the parameter values of multiple perspective transformation parameters of the first perspective transformation matrix between the first orthographic image and the second orthographic image . This process includes a first construction step, a second construction step, and a solution step, where,

[0115] First construction step: Construct a first perspective transformation matrix containing 8 unknown perspective transformation parameters

[0116]

[0117] where, , , , , , , and are the perspective transformation parameters in the first perspective transformation matrix .

[0118] Second construction step: Based on a feature point pair between the first orthographic image and the second orthographic image and , construct the mapping relationship between the first orthographic image and the second orthographic image , that is, there is

[0119] =

[0120] where s = + +1, for representing the pixel coordinates of the k-th feature point in the first orthographic image and for representing the abscissa of the k-th feature point in the image coordinate system of the first orthographic image and for representing the ordinate of the k-th feature point in the image coordinate system of the first orthographic image and for representing the pixel coordinates of the k-th feature point in the second orthographic image and for representing the abscissa of the k-th feature point in the image coordinate system of the second orthographic image and for representing the ordinate of the k-th feature point in the image coordinate system of the second orthographic image

[0121] Solution steps: Calculate = to obtain

[0122] ( )= + +

[0123] ( )= + +

[0124] After arrangement, we get

[0125]

[0126]

[0127] It can be represented by a matrix as

[0128] =

[0129] It can be simplified as: = , where A is used to represent and B is used to represent .

[0130] Since the first perspective transformation matrix constructed contains 8 perspective transformation parameters, in the first orthographic image and the second orthographic image ​​​​​​After there are at least 4 feature point pairs, the first perspective transformation matrix can be directly solved. , that is = , thus obtaining the first orthographic image and the second orthographic image and the first perspective transformation matrix between them. This first perspective transformation matrix can be used to represent the first mapping relationship between the first orthographic image and the second orthographic image .

[0131] Since the pixel coordinates of the feature point pairs between the first orthographic image and each second orthographic image are known and accurate, by using the feature point pairs between the first orthographic image and each second orthographic image to determine the parameter values of the multiple transformation parameters of the first transformation matrix, the parameter values of the multiple transformation parameters of the first transformation matrix can be accurately determined, so that the determined first transformation matrix can accurately reflect the mapping relationship between the first orthographic image and the corresponding second orthographic image.

[0132] In some embodiments, the second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system can be represented by a second transformation matrix, where the second transformation matrix includes multiple transformation parameters. In the following embodiments, taking perspective transformation as an example, the second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system is described, that is, using a second perspective transformation matrix including multiple perspective transformation parameters as an example of the second transformation matrix, but it can be understood that under the overall concept of this application, perspective transformation can be replaced with other geometric transformations according to needs.

[0133] As Figure 4 shown, another embodiment of this application provides a flowchart for determining the second mapping relationship based on perspective transformation. Figure 4 Different from other embodiments, step S205 in other embodiments can be specifically refined into Figure 4 steps S2051 to S2053 in

[0134] In step S2051, the geographic coordinates of each image center point are used as the geographic coordinates of the corresponding corresponding points.

[0135] Since the image center points of each second orthographic image and the corresponding points in the first orthographic image are in one-to-one correspondence, the geographic coordinates of the image center points of each second orthographic image can be directly determined as the geographic coordinates of the corresponding points.

[0136] In step S2052, a second perspective transformation matrix is constructed, and the second perspective transformation matrix contains multiple perspective transformation parameters.

[0137] The construction process of the second perspective transformation matrix is the same as that of the first perspective transformation matrix in the above example, and will not be elaborated here.

[0138] In step S2053, based on the second perspective transformation matrix, the geographical coordinates and pixel coordinates of each corresponding point, the parameter values of multiple perspective transformation parameters of the second perspective transformation matrix are determined, and the second perspective transformation matrix is used to represent the second mapping relationship.

[0139] Regarding the determination of the parameter values of multiple perspective transformation parameters of the second perspective transformation matrix, the process is the same as that of determining the parameter values of multiple perspective transformation parameters of the first perspective transformation matrix in the above example, and will not be elaborated here.

[0140] Since the geographical coordinates of each second orthographic image are known and accurate, therefore, determining the geographical coordinates of each second orthographic image as the geographical coordinates of the image center point of each second orthographic image can accurately determine the geographical coordinates of the image center point of each second orthographic image. Since each image center point and each corresponding point are in one-to-one correspondence, determining the geographical coordinates of each image center point as the geographical coordinates of each corresponding point can accurately determine the geographical coordinates of each corresponding point. Based on the geographical coordinates and pixel coordinates of each corresponding point, the parameter values of multiple transformation parameters of the second transformation matrix are determined, so that the parameter values of multiple transformation parameters in the second transformation matrix can be accurately determined, and thus the determined second mapping relationship can accurately reflect the mapping relationship between the image coordinate system and the geographical coordinate system of the first orthographic image.

[0141] In one embodiment, based on the second perspective transformation matrix, the geographical coordinates and pixel coordinates of the image center point of the first orthographic image, and the geographical coordinates and pixel coordinates of each corresponding point, the parameter values of multiple perspective transformation parameters of the second perspective transformation matrix are determined.

[0142] For example, use w to represent the height of the orthographic image and h to represent the width of the orthographic image, then the first orthographic image The pixel coordinates of the image center point can be expressed as . The pixel coordinates of the image center point and each corresponding point in the first orthographic image can be expressed as

[0143] =

[0144] Wherein, in ( , ) is used to represent the pixel coordinates of the image center point of the first orthographic image , in ( , ), which is used to represent the pixel coordinates of the \(j_m\)-th corresponding point in the first orthographic image , and is used to represent the abscissa of the \(j_m\)-th corresponding point in the image coordinate system of the first orthographic image , and is used to represent the ordinate of the \(j_m\)-th corresponding point in the image coordinate system of the first orthographic image . , and is used to represent the ordinate of the \(j_m\)-th corresponding point in the image coordinate system of the first orthographic image .

[0145] Since the geographical coordinates of the image center points of the second orthographic images are the geographical coordinates of the corresponding points in the first orthographic image, the geographical coordinates of the image center point and the corresponding points of the first orthographic image can be expressed as :

[0146]

[0147] wherein, the ( , ) is used to represent the geographical coordinates of the image center point in the first orthographic image, and the ( , , ) is used to represent the geographical coordinates of the \(j_m\)-th corresponding point in the first orthographic image, , is used to represent the abscissa of the \(j_m\)-th corresponding point in the geographical coordinate system, and is used to represent the ordinate of the \(j_m\)-th corresponding point in the geographical coordinate system.

[0148] Therefore, the parameter values of the perspective transformation parameters of the second perspective transformation matrix can be determined with reference to the process of determining the parameter values of the perspective transformation parameters of the first perspective transformation matrix in the above example, based on matrix and matrix .

[0149] In the actual application process, the corner points of an image can form the entire coordinate range of the image. Based on this, as Figure 5 shown, in another embodiment of the present application, a flowchart for determining the geographical range of the first orthographic image based on the corner points of the first orthographic image is provided. Figure 5 The difference from other embodiments is that step S206 in other embodiments can be specifically refined into Figure 5 step S2061 and step S2062 in

[0150] In step S2061, based on the pixel coordinates of the corner points of the first orthographic image and the second mapping relationship, the geographical coordinates of the corner points of the first orthographic image are obtained.

[0151] Each corner point of the first orthographic image may be a feature point. Of course, each corner point of the first orthographic image may also be a pixel point other than a feature point.

[0152] Each corner point of the first orthographic image may be a point where two mutually perpendicular sides of the first orthographic image intersect. Therefore, the number of corner points of the first orthographic image may be four.

[0153] In step S2062 , the geographical range of the first orthographic image is determined based on the geographical coordinates of the corner points of the first orthographic image.

[0154] Since the second mapping relationship is the mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image, the pixel coordinates of each corner point of the first orthographic image can be mapped based on the second mapping relationship, and the geographic coordinates of each corner point can be obtained quickly and accurately, so that the geographic range of the first orthographic image can be quickly determined based on the geographic coordinates of each corner point.

[0155] For example, if w is used to represent the height of the orthophoto image and h is used to represent the width of the orthophoto image, then the first orthophoto image The four corner points ( 、 、 as well as ) can be expressed as:

[0156] , , ,

[0157] If adopted Represents the first orthographic image The four corner points ( 、 、 as well as ) and the geographical coordinates of = express For the first orthographic image Corner point , then

[0158] =

[0159] in, Used to represent corner points The geographical coordinates of Used to represent corner points The horizontal coordinate in the geographic coordinate system, Used to represent corner points The vertical coordinate in the geographic coordinate system, For representing the second mapping relationship, related to the parameters of the last row of the s and matrix, and used for normalizing the homogeneous coordinate form.

[0160] Based on the second mapping relationship , respectively, for the diagonal points , and the corner points perform mapping, so as to obtain the corresponding , and , and then the four geographic coordinates of the first orthoimage can be respectively expressed as:

[0161] = , = , = , =

[0162] Among them, is used to represent the geographic coordinates of the corner point , is used to represent the abscissa of the corner point in the geographic coordinate system, is used to represent the ordinate of the corner point in the geographic coordinate system, is used to represent the geographic coordinates of the corner point , is used to represent the abscissa of the corner point in the geographic coordinate system, is used to represent the ordinate of the corner point in the geographic coordinate system, is used to represent the geographic coordinates of the corner point , is used to represent the abscissa of the corner point in the geographic coordinate system, is used to represent the ordinate of the corner point

[0163] From , and , the geographic range of the first orthoimage can be formed.

[0164] In addition, for the other pixel points in the first orthoimage except for the four corner points, the The geographical coordinates of other pixel points except the four corner points inside.

[0165] In some examples, the image processing method of the present application further includes: performing distortion correction on the first orthographic image and multiple second orthographic images to obtain the distortion-corrected first orthographic image and multiple second orthographic images, and the distortion-corrected first orthographic image and multiple second orthographic images are used to extract feature points.

[0166] By performing distortion correction on the first orthographic image and multiple second orthographic images, the geometric distortion introduced during the imaging process can be eliminated, and further, it can be ensured that the feature points of the distortion-corrected first orthographic image and multiple second orthographic images can be extracted more accurately subsequently.

[0167] In one embodiment, the first orthographic image and multiple second orthographic images can be subjected to distortion correction through the following steps. The specific process includes a first determination step, a distortion correction step, and a distortion correction processing step, where

[0168] First determination step: Determine the camera internal parameter matrix K, and the camera internal parameter matrix K can be expressed as

[0169]

[0170] Where Is used to represent the effective focal length of the first orthographic image or each second orthographic image on the x-axis, Is used to represent the effective focal length of the first orthographic image or each second orthographic image on the y-axis, And Are the principal point coordinates, that is Is used to represent the abscissa of the image center point in the pixel coordinate system, Is used to represent the ordinate of the image center point in the pixel coordinate system.

[0171] Distortion correction step: The image distortion correction coefficients usually include radial distortion coefficients and tangential distortion coefficients. Among them, the radial distortion coefficients include , And , the tangential distortion coefficients include And . Based on this, the distortion model of the first orthographic image or each second orthographic image can be expressed as

[0172] (1 + + ) + 2 xy + )

[0173] (1 + + )+ xy+ )

[0174] Among them, ( ) is used to represent the pixel coordinates of the pixel points in the first orthographic image or each second orthographic image after distortion correction, is used to represent the abscissa of the pixel points in the first orthographic image or each second orthographic image after distortion correction, is used to represent the ordinate of the pixel points in the first orthographic image or each second orthographic image after distortion correction, (x, y) is the pixel coordinates of the pixel points in the first orthographic image or each second orthographic image before distortion correction, x is used to represent the abscissa of the pixel points in the first orthographic image or each second orthographic image before distortion correction, and y is used to represent the ordinate of the pixel points in the first orthographic image or each second orthographic image before distortion correction, = .

[0175] Distortion correction processing steps: Construct a mapping matrix, and perform iterative interpolation on the pixel coordinates of the pixel points in the first orthographic image or each second orthographic image before distortion correction and the corresponding pixel coordinates of the pixel points after distortion correction in the distortion correction step, so as to obtain the mapping matrix between the first orthographic image or each second orthographic image before and after distortion correction, that is, obtain the mapping table of the distorted coordinates and undistorted coordinates of the first orthographic image or each second orthographic image. Then, based on the mapping table, map the pixel points in the first orthographic image or each second orthographic image one by one to obtain the corresponding image after distortion correction.

[0176] For the sake of easy understanding, the following takes the inspection of a photovoltaic power station by a drone as an example to illustrate the processing process of the image processing method of the present application. Figure 6 This is a flowchart of the image processing method for the inspection of a photovoltaic power station by a drone provided in an embodiment of the present application. The image processing method for the inspection of a photovoltaic power station by a drone includes steps S601 to S608.

[0177] In step S601, the drone inspects the photovoltaic power station and takes n orthographic images, that is, the orthographic image set .

[0178] In step S602, determine the first orthographic image and multiple second orthographic images from the multiple orthographic images.

[0179] When estimating the geographical range, is used as the first orthographic image. Determine and the geographical distance between them, and determine the orthophoto image whose geographical distance meets the preset condition as the second orthophoto image. For the convenience of understanding, assume that and the geographical distance between each orthophoto image in meets the preset condition, then the orthophoto images in

[0180] In step S603, perform distortion correction on the first orthophoto image and multiple second orthophoto images.

[0181] In step S604, perform image matching on the first orthophoto image and each second orthophoto image to determine the feature point pairs between the first orthophoto image and each second orthophoto image.

[0182] Adopt the SIFT algorithm to extract the feature points of the first orthophoto image and each second orthophoto image. Based on the feature vector distance between the feature points of the first orthophoto image and the feature points of each second orthophoto image, perform image matching on the first orthophoto image and each second orthophoto image to determine the feature point pairs between the first orthophoto image and each second orthophoto image.

[0183] In step S605, based on the feature point pairs between the first orthophoto image and each second orthophoto image, determine the first mapping relationship on the image between the first orthophoto image and each second orthophoto image.

[0184] In step S606, based on the first mapping relationship, in the first orthophoto image, determine the corresponding point corresponding to the image center point of the second orthophoto image.

[0185] In step S607, based on the pixel coordinates and geographical coordinates of the image center point of the first orthophoto image, and the pixel coordinates and geographical coordinates of the corresponding point, determine the second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthophoto image.

[0186] Since the first orthophoto image and each second orthophoto image are taken when the lens of the image acquisition device configured on the drone is vertically downward facing the ground, therefore, the geographical coordinates of the first orthophoto image can be determined as the geographical coordinates of the image center point of the first orthophoto image, and the geographical coordinates of each second orthophoto image can be determined as the geographical coordinates of the image center point of each second orthophoto image.

[0187] Since each corresponding point in the first orthophoto image has a one-to-one correspondence with the image center point of each second orthophoto image, the geographical coordinates of the image center point of each second orthophoto image can be used as the geographical coordinates of each corresponding point in the first orthophoto image.

[0188] Based on this, the geographical coordinates of the image center point and each corresponding point in the first orthophoto image can be obtained.

[0189] The determination process of the geographical coordinates of the image center point of the first orthophoto image and the image center points of each second orthophoto image is as follows:

[0190] From the EXIF files of the first orthophoto image and each second orthophoto image, determine the longitude and latitude parameters of the first orthophoto image and each second orthophoto image, which can be specifically expressed as

[0191] LON = { }

[0192] LAT = { }

[0193] Among them, the LON set is the longitude parameter set, and among the elements of the LON set is used to represent the longitude parameter of the nth orthophoto image. The LAT set is the latitude parameter set, and among the LAT set is used to represent the latitude parameter of the nth orthophoto image.

[0194] Based on the longitude and latitude parameters of the first orthophoto image and each second orthophoto image, determine the geographical coordinates of the first orthophoto image and each second orthophoto image. It can be specifically expressed as:

[0195] LX =

[0196] LY = { }

[0197] Among them, the LX set is the abscissa set, and in the LX set is used to represent the abscissa of the nth orthophoto image in the geographical coordinate system. The LY set is the ordinate set, and in the LY set is used to represent the ordinate of the nth orthophoto image in the geographical coordinate system.

[0198] In step S608, based on the pixel coordinates of the positioning point of the first orthophoto image and the second mapping relationship, determine the geographical range of the first orthophoto image.

[0199] Based on the same inventive concept, an embodiment of the present application also provides an image processing device. Specifically combined with Figure 7 a detailed description of the image processing device provided by the embodiment of the present application is given.

[0200] Figure 7 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.

[0201] As Figure 7As shown in the figure, the image processing device 700 may include an acquisition unit 710, a matching unit 720, a first determination unit 730, a second determination unit 740, a third determination unit 750, and a fourth determination unit 760.

[0202] The acquisition unit 710 is configured to acquire a first orthographic image and multiple second orthographic images, and there is an image overlapping area between the first orthographic image and each second orthographic image.

[0203] The matching unit 720 is configured to perform image matching on the first orthographic image and each second orthographic image, and determine the matching feature point pairs between the first orthographic image and each second orthographic image.

[0204] The first determination unit 730 is configured to respectively determine the first mapping relationship on the image between the first orthographic image and each second orthographic image based on the matching feature point pairs between the first orthographic image and each second orthographic image.

[0205] The second determination unit 740 is configured to determine the pixel coordinates of each corresponding point in the first orthographic image based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center point of each second orthographic image, and each corresponding point corresponds one-to-one to the image center point of each second orthographic image.

[0206] The third determination unit 750 is configured to determine the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point.

[0207] The fourth determination unit 760 is configured to determine the geographic range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

[0208] In one embodiment, the first determination unit is configured to construct a first transformation matrix, and the first transformation matrix contains multiple transformation parameters; for the first orthographic image and any second orthographic image, based on the first transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any second orthographic image, determine the parameter values of the multiple transformation parameters of the first transformation matrix, and the first transformation matrix is used to represent the first mapping relationship.

[0209] The third determination unit is configured to determine the geographic coordinates of each second orthographic image based on the longitude and latitude parameters of each second orthographic image; and determine the geographic coordinates of the image center point of each second orthographic image as the geographic coordinates of each second orthographic image.

[0210] The third determination unit may also be configured to use the geographical coordinates of the center points of the images as the geographical coordinates of the corresponding points; construct a second transformation matrix, where the second transformation matrix contains multiple transformation parameters; based on the second transformation matrix, the geographical coordinates and pixel coordinates of the corresponding points, determine the parameter values of the multiple transformation parameters of the second transformation matrix, and the second transformation matrix is used to represent the second mapping relationship.

[0211] In one embodiment, the fourth determination unit may also be configured to, when the positioning point is a corner point of the first orthographic image, based on the pixel coordinates of the corner points of the first orthographic image and the second mapping relationship, obtain the geographical coordinates of the corner points of the first orthographic image; based on the geographical coordinates of the corner points of the first orthographic image, determine the geographical range of the first orthographic image.

[0212] In one embodiment, the acquisition unit may also be configured to screen out multiple second orthographic images with image overlapping regions with the first orthographic image from multiple orthographic images.

[0213] The acquisition unit may also be configured to determine the geographical distances between the first orthographic image and each orthographic image based on the geographical coordinates of the first orthographic image and each orthographic image; determine whether the geographical distances meet a preset condition; when the geographical distances meet the preset condition, determine the orthographic image as the second orthographic image.

[0214] In one embodiment, the image processing device further includes a distortion correction unit. The distortion correction unit may also be configured to perform distortion correction on the first orthographic image and multiple second orthographic images to obtain the distortion-corrected first orthographic image and multiple second orthographic images, and the distortion-corrected first orthographic image and multiple second orthographic images are used to extract feature points.

[0215] It should be noted that this image processing device corresponds to the above image processing method. All implementation manners in the above method embodiments are applicable to the embodiments of this device and can also achieve the same technical effects, which will not be elaborated here.

[0216] Figure 8 FIG. shows a schematic hardware structure diagram of an electronic device provided in an embodiment of the present application.

[0217] The electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0218] Specifically, the above-mentioned processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be an integrated circuit configured to implement one or more embodiments of the present application.

[0219] The memory 802 may include a mass storage for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 802 may include removable or non-removable (or fixed) media. Where appropriate, the memory 802 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, the memory 802 is a non-volatile solid-state memory.

[0220] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present application.

[0221] The processor 801 reads and executes the computer program instructions stored in the memory 802 to implement any one of the image processing methods in the above embodiments.

[0222] In one example, the electronic device may further include a communication interface 803 and a bus 810. Among them, as Figure 8 shown, the processor 801, the memory 802, and the communication interface 803 are connected through the bus 810 to complete communication with each other.

[0223] The communication interface 803 is mainly used to implement communication between the various modules, devices, units, and / or devices in the embodiments of the present application.

[0224] The bus 810 includes hardware, software, or both, and couples components together. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 810 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0225] In addition, in combination with the image processing method in the above embodiments, the embodiments of the present application may provide a computer storage medium to implement. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the image processing methods in the above embodiments is implemented.

[0226] The embodiments of the present application also provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the image processing method provided by the embodiments of the present application.

[0227] The embodiments of the present application also provide an electronic device. The electronic device includes: a processor and a memory storing computer program instructions, the processor being coupled to the memory; when the processor executes the computer program instructions, the image processing method provided by the embodiments of the present application is implemented.

[0228] It should be clear that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0229] The functional blocks shown in the above-described structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments for performing the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted via a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0230] It should also be noted that in the exemplary embodiments mentioned in the present application, some methods or systems are described based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0231] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each block in the flowcharts and / or block diagrams, and the combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general purpose processor, a special purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagrams and / or flowcharts, and the combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware for performing the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0232] The above are only specific embodiments of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present application.

Claims

1. An image processing method, characterized in that, Including: Obtain a first orthophoto image and multiple second orthophoto images. There is an image overlapping area between the first orthophoto image and each of the second orthophoto images. The obtaining of the first orthophoto image and the multiple second orthophoto images includes: obtaining a set of orthophoto images collected by an image acquisition device during the inspection process of a drone; for the first orthophoto image in the set of orthophoto images, determine a second orthophoto image in the set of orthophoto images that has an image overlapping area with the first orthophoto image, and there is a geographical distance between the second orthophoto image and the first orthophoto image. Perform image matching on the first orthophoto image and each of the second orthophoto images to determine pairs of matching feature points between the first orthophoto image and each of the second orthophoto images. Based on the pixel coordinates of the pairs of matching feature points between the first orthophoto image and each of the second orthophoto images, respectively determine a first mapping relationship on the image between the first orthophoto image and each of the second orthophoto images. The first mapping relationship is used to determine the geometric transformation relationship between the first orthophoto image and each of the second orthophoto images, and the first mapping relationship is a first perspective transformation matrix. Based on the first mapping relationship corresponding to each of the second orthophoto images and the pixel coordinates of the image center points of each of the second orthophoto images, determine the pixel coordinates of the corresponding points in the first orthophoto image. Each of the corresponding points corresponds one-to-one to the image center point of each of the second orthophoto images. Based on the pixel coordinates of each of the corresponding points in the first orthophoto image and the geographical coordinates of the image center points of each of the second orthophoto images, determine a second mapping relationship between the image coordinate system and the geographical coordinate system of the first orthophoto image. Based on the pixel coordinates of the positioning points of the first orthophoto image and the second mapping relationship, determine the geographical range of the first orthophoto image. Among them, the process of determining the geographical coordinates of the image center points of each of the second orthophoto images includes: determining the longitude and latitude parameters of each of the second orthophoto images based on the shooting conditions recorded by the image acquisition device, and based on the longitude and latitude parameters of each of the second orthophoto images, determining the geographical coordinates of each of the second orthophoto images; determining the geographical coordinates of each of the second orthophoto images as the geographical coordinates of the image center points of each of the second orthophoto images.

2. The method according to claim 1, characterized in that Respectively determining the first mapping relationship on the image between the first orthophoto image and each of the second orthophoto images includes: Construct a first perspective transformation matrix, and the first perspective transformation matrix contains multiple transformation parameters. For the first orthophoto image and any one of the second orthophoto images, based on the first perspective transformation matrix and the pixel coordinates of the pair of matching feature points between the first orthophoto image and any one of the second orthophoto images, determine the parameter values of the multiple transformation parameters of the first perspective transformation matrix. The first perspective transformation matrix is used to represent the first mapping relationship.

3. The method according to claim 1, wherein Determining a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthophoto image based on the pixel coordinates of the corresponding points in the first orthophoto image and the geographic coordinates of the image center points of the second orthophoto images, includes: Regarding the geographic coordinates of the image center points as the geographic coordinates of the corresponding points; Constructing a second transformation matrix, which contains multiple transformation parameters; Based on the second transformation matrix, the geographic coordinates and pixel coordinates of the corresponding points, determining the parameter values of the multiple transformation parameters of the second transformation matrix, where the second transformation matrix is used to represent the second mapping relationship.

4. The method according to claim 1, wherein The positioning points are at least three pixel points not on a straight line.

5. The method according to claim 4, wherein In the case where the positioning points are the corner points of the first orthophoto image, Determining the geographic range of the first orthophoto image based on the pixel coordinates of the positioning points of the first orthophoto image and the second mapping relationship, includes: Obtaining the geographic coordinates of the corner points of the first orthophoto image based on the pixel coordinates of the corner points of the first orthophoto image and the second mapping relationship; Determining the geographic range of the first orthophoto image based on the geographic coordinates of the corner points of the first orthophoto image.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining the first orthophoto image and multiple second orthophoto images includes: Selecting multiple second orthophoto images with image overlapping regions with the first orthophoto image from multiple orthophoto images.

7. The method according to claim 6, wherein The selecting multiple second orthophoto images with image overlapping regions with the first orthophoto image from multiple orthophoto images includes: Determining the geographic distance between the first orthophoto image and each orthophoto image based on the geographic coordinates of the first orthophoto image and each orthophoto image; Determining whether the geographic distance meets a preset condition; In the case where the geographic distance meets the preset condition, determining the orthophoto image as the second orthophoto image.

8. The method according to any one of claims 1 to 5, characterized in that The method further includes: Performing distortion correction on the first orthophoto image and multiple second orthophoto images to obtain the distortion-corrected first orthophoto image and multiple second orthophoto images, where the distortion-corrected first orthophoto image and multiple second orthophoto images are used to extract the feature points.

9. An image processing apparatus, characterized in that, Includes: An obtaining unit, configured to obtain a first orthophoto image and multiple second orthophoto images, where the first orthophoto image and each second orthophoto image have an image overlapping region, and the obtaining the first orthophoto image and multiple second orthophoto images includes: obtaining a set of orthophoto images collected by an image acquisition device during the inspection of a drone; for the first orthophoto image in the set of orthophoto images, determining a second orthophoto image with an image overlapping region with the first orthophoto image from the set of orthophoto images, where there is a geographic distance between the second orthophoto image and the first orthophoto image; A matching unit, configured to perform image matching on the first orthophoto image and each second orthophoto image to determine the pairs of matching feature points between the first orthophoto image and each second orthophoto image; A first determination unit, configured to respectively determine a first mapping relationship on the image between the first orthographic image and each of the second orthographic images based on pixel coordinates of the feature point pairs that match between the first orthographic image and each of the second orthographic images, where the first mapping relationship is used to determine a geometric transformation relationship between the first orthographic image and each of the second orthographic images, and the first mapping relationship is a first perspective transformation matrix; A second determination unit, configured to determine pixel coordinates of corresponding points in the first orthographic image based on the first mapping relationship corresponding to each of the second orthographic images and pixel coordinates of the image center points of each of the second orthographic images, where each of the corresponding points corresponds one-to-one to the image center point of each of the second orthographic images; A third determination unit, configured to determine a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image based on pixel coordinates of each of the corresponding points in the first orthographic image and geographic coordinates of the image center points of each of the second orthographic images; A fourth determination unit, configured to determine a geographic range of the first orthographic image based on pixel coordinates of a positioning point of the first orthographic image and the second mapping relationship; Wherein, the process of determining geographic coordinates of the image center points of each of the second orthographic images includes: determining longitude and latitude parameters of each of the second orthographic images based on shooting conditions recorded by the image acquisition device, and determining geographic coordinates of each of the second orthographic images based on the longitude and latitude parameters of each of the second orthographic images; and determining geographic coordinates of the image center points of each of the second orthographic images as the geographic coordinates of each of the second orthographic images.

10. A computer-readable storage medium, characterized in that, A computer program instruction is stored on the computer-readable storage medium, and when the computer program instruction is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

11. A computer program product, characterized in that, When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the image processing method according to any one of claims 1 to 8.

12. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions, where the processor is coupled to the memory; and when the processor executes the computer program instructions, the image processing method according to any one of claims 1 to 8 is implemented.

Images