Image processing method and device, storage medium, program product and electronic equipment

Abstract

The invention discloses an image processing method and device, a storage medium, a program product and electronic equipment. The method comprises the following steps: determining matched feature point pairs between a first orthographic image and each second orthographic image; based on the matched feature point pairs between the first orthographic image and each second orthographic image, respectively determining a first mapping relationship between the first orthographic image and each second orthographic image on the image; based on a first mapping relation corresponding to each second orthographic image and a pixel coordinate of an image center point of each second orthographic image, determining a pixel coordinate of each corresponding point in the first orthographic image; determining a second mapping relation between the image coordinate system of the first orthographic image and the geographic coordinate system based on the pixel coordinates of the corresponding points and the geographic coordinates of the image center points; and determining the geographical range of the first orthographic image based on the pixel coordinates of the positioning points of the first orthographic image and the second mapping relation, thereby accurately determining the geographical range of the first orthographic image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, device, storage medium, program product and electronic device. Background Art

[0002] With the widespread promotion of new energy technologies, photovoltaic power generation, as a representative of clean energy, has led to the rapid development of photovoltaic power stations. At present, the daily management and maintenance of photovoltaic power stations are usually carried out based on the images taken of photovoltaic power stations.

[0003] However, it is currently impossible to quickly and accurately estimate the actual geographical scope of the images of the photovoltaic power station, which directly affects the subsequent key links such as image stitching, three-dimensional modeling, and accurate combing of image information, and causes certain difficulties in the daily management and maintenance of photovoltaic power stations. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, storage medium, program product, and electronic device, which can quickly and accurately estimate the geographical range corresponding to an orthographic image.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, including: acquiring a first orthographic image and multiple second orthographic images, wherein the first orthographic image and each second orthographic image have an image overlapping area; performing image matching on the first orthographic image and each second orthographic image to determine matching feature point pairs between the first orthographic image and each second orthographic image; based on the matching feature point pairs between the first orthographic image and each second orthographic image, respectively determining a first mapping relationship between the first orthographic image and each second orthographic image on the image; based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center point of each second orthographic image, determining the pixel coordinates of each corresponding point in the first orthographic image, wherein each corresponding point corresponds one-to-one with the image center point of each second orthographic image; based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point, determining a second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system; based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship, determining the geographic range of the first orthographic image.

[0006] In an optional implementation of the first aspect, a first mapping relationship between a first orthographic image and each second orthographic image on an image is determined respectively, including: constructing a first transformation matrix, the first transformation matrix containing a plurality of transformation parameters; for the first orthographic image and any second orthographic image, based on the first transformation matrix and the pixel coordinates of a feature point pair between the first orthographic image and any second orthographic image, determining parameter values ​​of a plurality of transformation parameters of the first transformation matrix, the first transformation matrix being used to characterize the first mapping relationship.

[0007] In an optional implementation of the first aspect, the process of determining the geographic coordinates of the image center point of each second orthophoto image includes: determining the geographic coordinates of each second orthophoto image based on the latitude and longitude parameters of each second orthophoto image; and determining the geographic coordinates of each second orthophoto image as the geographic coordinates of the image center point of each second orthophoto image.

[0008] In an optional implementation of the first aspect, based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point, a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image is determined, including: taking the geographic coordinates of each image center point as the corresponding geographic coordinates of each corresponding point; constructing a second transformation matrix, the second transformation matrix containing multiple transformation parameters; based on the second transformation matrix, the geographic coordinates and pixel coordinates of each corresponding point, determining parameter values ​​of multiple transformation parameters of the second transformation matrix, the second transformation matrix being used to characterize the second mapping relationship.

[0009] In an optional implementation of the first aspect, the positioning points are at least three pixel points that are not on a straight line.

[0010] In an optional implementation of the first aspect, when the positioning point is a corner point of the first orthographic image, the geographic scope of the first orthographic image is determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship, including: obtaining the geographic coordinates of each corner point of the first orthographic image based on the pixel coordinates of each corner point of the first orthographic image and the second mapping relationship; determining the geographic scope of the first orthographic image based on the geographic coordinates of each corner point of the first orthographic image.

[0011] In an optional implementation of the first aspect, acquiring the first orthographic image and the plurality of second orthographic images includes: screening out, from the plurality of orthographic images, the plurality of second orthographic images having image overlapping areas with the first orthographic image.

[0012] In an optional implementation of the first aspect, a plurality of second orthographic images having image overlapping areas with the first orthographic image are screened out from a plurality of orthographic images, including: determining a geographical distance between the first orthographic image and each orthographic image based on the geographical coordinates of the first orthographic image and each orthographic image; determining whether the geographical distance satisfies a preset condition; and determining the orthographic image as the second orthographic image when the geographical distance satisfies the preset condition.

[0013] In an optional implementation of the first aspect, the image processing method further includes: performing distortion correction on the first orthographic image and the multiple second orthographic images to obtain the distortion-corrected first orthographic image and the multiple second orthographic images, and the distortion-corrected first orthographic image and the multiple second orthographic images are used to extract feature points.

[0014] In a second aspect, an embodiment of the present application provides an image processing device, including: an acquisition unit, used to acquire a first orthographic image and multiple second orthographic images, wherein the first orthographic image and each second orthographic image have an image overlapping area; a matching unit, used to perform image matching on the first orthographic image and each second orthographic image, and determine the feature point pairs that match between the first orthographic image and each second orthographic image; a first determination unit, used to determine the first mapping relationship between the first orthographic image and each second orthographic image on the image based on the feature point pairs that match between the first orthographic image and each second orthographic image; a second determination unit, used to determine the pixel coordinates of each corresponding point in the first orthographic image based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center point of each second orthographic image, wherein each corresponding point corresponds one-to-one with the image center point of each second orthographic image; a third determination unit, used to determine the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point; a fourth determination unit, used to determine the geographic range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

[0015] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the image processing method of the first aspect is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image processing method of the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides an electronic device. The electronic device includes a processor and a memory storing computer program instructions, the processor is coupled to the memory, and when the processor executes the computer program instructions, the image processing method of the first aspect is implemented.

[0018] In the embodiment of the present application, since the first orthographic image and each second orthographic image have an image overlap area, image matching is performed on the first orthographic image and multiple second orthographic images, and feature point pairs having a mapping relationship between the first orthographic image and each second orthographic image can be determined. By using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, that is, the first mapping relationship, can be accurately and efficiently determined, so that by using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped to the first orthographic image, and the corresponding point corresponding to the image center point is determined in the first orthographic image. Since the geographic coordinates of the image center point of each second orthographic image are known accurately, and the image center point of each second orthographic image corresponds one-to-one to the corresponding point in the first orthographic image, the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image can be accurately determined based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point. Since the second mapping relationship accurately describes the mapping relationship between the image space of the first orthographic image and the physical space, the geographical scope of the first orthographic image can be quickly and accurately determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship. It can be seen that in the process of determining the geographical scope of the first orthographic image, the present application cleverly utilizes the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the center point of the image to quickly and accurately determine the geographical scope of the first orthographic image. On this basis, the accurately estimated geographical scope can be used to stitch and process multiple images of the photovoltaic power station, analyze the overall area of ​​the photovoltaic power station and locate problems, and effectively improve the daily management and maintenance efficiency of the photovoltaic power station. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a schematic diagram of the architecture of an image processing system provided by an embodiment of the present application; Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application; Figure 3 is a flowchart of determining a first mapping relationship based on perspective transformation provided by an embodiment of the present application; Figure 4 is a flowchart of determining a second mapping relationship based on perspective transformation provided by an embodiment of the present application; Figure 5is a flow chart for determining the geographic range of a first orthographic image provided by an embodiment of the present application; Figure 6 This is a flow chart of an image processing method for a scenario in which a drone inspects a photovoltaic power station, provided by an embodiment of the present application; Figure 7 is a structural schematic diagram of an image processing device provided by another embodiment of the present application; Figure 8 It is a structural diagram of an electronic device provided by yet another embodiment of the present application.

[0021] The above drawings include the following reference numerals: 100, electronic device; 200, drone; 700, image processing device; 710, acquisition unit; 720, matching unit; 730, first determination unit; 740, second determination unit; 750, third determination unit; 760, fourth determination unit; 801, processor; 802, memory; 803, communication interface; 810, bus. DETAILED DESCRIPTION

[0022] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0023] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0024] With the continuous expansion of renewable energy technology, photovoltaic power stations are being built in an all-round way. Usually, the daily management and maintenance of photovoltaic power stations are carried out based on the images of photovoltaic power stations taken. However, it is currently impossible to quickly and accurately estimate the actual geographical scope corresponding to the images of photovoltaic power stations taken, resulting in the inability to use the accurately estimated geographical scope to stitch multiple images of photovoltaic power stations taken, three-dimensional modeling, and accurate combing of image information, etc., which has caused certain difficulties in analyzing the overall area of ​​photovoltaic power stations and locating problems, and thus leads to low efficiency in daily management and maintenance of photovoltaic power stations. For example, in the scenario where drones inspect photovoltaic power stations, the image acquisition equipment configured by the drones can capture images of photovoltaic power stations on the inspection route. However, since it is impossible to quickly and accurately estimate the geographical scope corresponding to each image, it is impossible to use the accurately estimated geographical scope to stitch the images of photovoltaic power stations taken by drones, thereby failing to accurately analyze the overall area of ​​photovoltaic power stations and locate problems.

[0025] The present application provides an image processing method, device, storage medium, program product and electronic device. Since the first orthographic image and each second orthographic image have an image overlapping area, the first orthographic image and the plurality of second orthographic images are image matched, and the feature point pairs having a mapping relationship between the first orthographic image and each second orthographic image can be determined. By using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, that is, the first mapping relationship, can be accurately and efficiently determined, so that by using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped to the first orthographic image, and the corresponding point corresponding to the image center point is determined in the first orthographic image. Since the geographic coordinates of the image center point of each second orthographic image are known accurately, and the image center point of each second orthographic image corresponds one-to-one to the corresponding point in the first orthographic image, the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image can be accurately determined based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point. Since the second mapping relationship accurately describes the mapping relationship between the image space of the first orthographic image and the physical space, the geographical scope of the first orthographic image can be quickly and accurately determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship. It can be seen that in the process of determining the geographical scope of the first orthographic image, the present application cleverly utilizes the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the center point of the image to quickly and accurately determine the geographical scope of the first orthographic image. On this basis, the accurately estimated geographical scope can be used to stitch and process multiple images of the photovoltaic power station, analyze the overall area of ​​the photovoltaic power station and locate problems, and effectively improve the daily management and maintenance efficiency of the photovoltaic power station.

[0026] For ease of understanding, here we take the example of drone inspection of photovoltaic power stations to briefly introduce the application architecture of the image processing method, device, storage medium, program product and electronic device provided in this application. Figure 1 A schematic diagram of the application architecture of an image processing system provided for an embodiment of the present application. The image processing system includes a drone 200 and an electronic device 100. The drone 200 inspects the photovoltaic power station based on the inspection route, and uses the image acquisition device configured for the drone to capture multiple images. It should be noted that when capturing multiple images, the image acquisition device of the drone keeps the lens pointed vertically downward at the ground to ensure that the multiple images captured are orthographic images. After receiving the multiple orthographic images captured by the drone 200, the electronic device 100 processes the multiple orthographic images according to the image processing method provided in the present application to determine the geographical range corresponding to each orthographic image.

[0027] In practical applications, in the embodiments of the present application, the image processing method provided in the embodiments of the present application can be implemented when the processor of the electronic device executes a program or instruction. However, in some embodiments, the image acquisition device of the drone or other devices may also have similar functions. For example, the image processing method provided in the embodiments of the present application is implemented when the image acquisition device of the drone executes a program or instruction. The embodiments of the present application are not limited to this.

[0028] It should be noted that the application scenarios described in the above embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. A person of ordinary skill in the art will know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. The image processing method provided by the embodiments of the present application can be applied to various application scenarios that require accurate estimation of the geographic range of an image.

[0029] The image processing method, device, storage medium, program product and electronic device provided by the present application are described below respectively.

[0030] Figure 2 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 As shown, the image processing method provided in the embodiment of the present application may include steps S201 to S206.

[0031] In step S201 , a first orthographic image and a plurality of second orthographic images are acquired, wherein the first orthographic image and each of the second orthographic images have an image overlapping area.

[0032] The first orthographic image and the plurality of second orthographic images are taken when the lens of the image acquisition device of the drone is pointed vertically downward at the ground.

[0033] The image overlap region refers to a region having the same or similar content between the first orthographic image and each second orthographic image. Optionally, the image overlap between the first orthographic image and each second orthographic image may be a complete overlap or a partial overlap.

[0034] In some embodiments, any one of the multiple orthographic images can be determined as the first orthographic image, and then, from the orthographic images other than the first orthographic image, multiple orthographic images having image overlapping areas with the first orthographic image are screened out as multiple second orthographic images.

[0035] Take the inspection of photovoltaic power stations by drones as an example. In actual applications, when drones perform inspection tasks, the image acquisition equipment configured by the drones takes pictures of the photovoltaic power station and obtains an orthographic image set containing n orthographic images. For example, in estimating When the geographical scope is As the first orthographic image, and can be obtained from { } There are multiple second orthographic images with overlapping image areas. When the geographical scope is As the first orthographic image, it can be obtained from Determine and There are a plurality of second orthographic images with overlapping image regions.

[0036] Since there are image overlapping areas between the first orthographic image and the plurality of second orthographic images, it is ensured that there are more common features between the first orthographic image and each of the second orthographic images, so feature point pairs can be accurately determined between the first orthographic image and each of the second orthographic images subsequently.

[0037] There are many ways to implement screening out a plurality of second orthographic images having image overlapping areas with the first orthographic image from a plurality of orthographic images.

[0038] In one embodiment, a neural network model may be used to perform target detection on a first orthographic image and multiple orthographic images, and then determine an intersection over union (IOU) value of a rectangular box (Bounding Box) obtained by target detection in the first orthographic image and a rectangular box (Bounding Box) obtained by target detection in each orthographic image. Finally, based on the IOU values ​​of the first orthographic image and each orthographic image, the orthographic image corresponding to an IOU value greater than a preset threshold is determined as the second orthographic image.

[0039] In another embodiment, the second orthographic image having an image overlapping area with the first orthographic image can be screened based on continuous shooting time screening. For example, an orthographic image whose shooting time interval with the first orthographic image is less than a preset threshold is determined as the second orthographic image.

[0040] In another embodiment, a second orthographic image having an image overlap area with the first orthographic image may be determined based on the geographic distance between the first orthographic image and each orthographic image. Specifically, based on the geographic coordinates of the first orthographic image and each orthographic image, the geographic distance between the first orthographic image and each orthographic image is determined; whether the geographic distance meets a preset condition is determined; and if the geographic distance meets the preset condition, the orthographic image is determined as the second orthographic image.

[0041] Since the geographic coordinates of the first orthographic image and each orthographic image are two-dimensional plane coordinates, we can use , determine the geographical distance between the first orthographic image and each orthographic image, where ( , ) is used to represent the geographic coordinates of the first orthographic image, It is used to represent the horizontal coordinate of the first orthographic image in the geographic coordinate system. Used to represent the vertical coordinate of the first orthographic image in the geographic coordinate system, ( , ) is used to represent the geographic coordinates of the orthographic image, Used to represent the horizontal coordinate of the orthographic image in the geographic coordinate system. Used to represent the vertical coordinate of the orthographic image in the geographic coordinate system.

[0042] The preset condition may be that the geographical distance between the first orthographic image and each orthographic image is less than a distance threshold. The distance threshold may be set based on the shooting height of the image acquisition device, the shooting angle of the image acquisition device, and the focal length of the image acquisition device.

[0043] Based on the geographical distance between the first orthographic image and each orthographic image, each orthographic image is screened, so that the second orthographic image having an image overlap area with the first orthographic image can be quickly determined.

[0044] In step S202, image matching is performed on the first orthographic image and each second orthographic image to determine matching feature point pairs between the first orthographic image and each second orthographic image.

[0045] Image matching is to determine the feature points of similar parts between different images and establish the correspondence between the feature points of similar parts between different images. Usually, image matching between two images can be performed based on feature point matching, region matching or global matching.

[0046] In one embodiment, the image matching is further explained by taking the inspection of a photovoltaic power station by a drone as an example. First, taking a first orthographic image and a second orthographic image as an example, feature points in the first orthographic image and feature points in the second orthographic image are extracted. Afterwards, based on the feature vector distance between the feature points in the first orthographic image and the feature points in the second orthographic image, it is determined whether the feature points in the first orthographic image and the feature points in the second orthographic image correspond to the same feature. If the feature vector distance between the feature points in the first orthographic image and the feature points in the second orthographic image meets the requirements, that is, the two feature points correspond to the same feature, then there is a corresponding relationship between the feature points in the first orthographic image and the feature points in the second orthographic image, so that the feature points in the first orthographic image and the feature points in the second orthographic image are determined as a feature point pair.

[0047] A feature point pair is two feature points with corresponding relationship in two images.

[0048] In actual application, any feasible feature extraction method in the prior art can be used to extract feature points in the first orthographic image or the second orthographic image. For example, algorithms such as Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), or Binary Robust Invariant Scalable Keypoints (BRISK) can be used to extract feature points in the first orthographic image or the second orthographic image.

[0049] In an optional embodiment, taking the SIFT algorithm as an example, the process of performing image matching on the first orthographic image and each second orthographic image to determine the feature point pairs between the first orthographic image and each second orthographic image is further described. The process includes a feature extraction step, a feature matching step, a feature matching result screening step, and a false matching removal step.

[0050] Feature extraction step: Extract SIFT feature points of the first orthographic image and each second orthographic image to obtain a feature point set SIFT={ , , , …, }, each element in the feature point set SIFT is used to represent a SIFT feature point in the first orthographic image or the second orthographic image, which is in the form of, for example but not limited to, a feature vector.

[0051] Feature matching step: using the nearest neighbor matching method, two feature points whose feature vectors are closest to the feature points in the first orthographic image are determined from each second orthographic image.

[0052] Feature matching result screening steps: For ease of understanding, a feature point in the first orthographic image and two feature points with the closest feature vector distance in the corresponding second orthographic image are used for illustration. , filter the feature vector distance between a feature point in the first orthophoto and two feature points in the corresponding second orthophoto, and filter out the feature vectors whose distance is greater than the feature threshold At the same time, determine the ratio of the feature vector distance between a feature point in the first orthographic image and the nearest feature point in the second orthographic image to the feature vector distance between a feature point in the first orthographic image and the second nearest feature point in the second orthographic image. In the case of , a feature point in the first orthographic image and the closest feature point in the second orthographic image are determined as a feature point pair. For example, the ratio threshold Can be set to 0.75.

[0053] Steps to remove mismatches: Use the Random Sample Consensus (RANSAC) algorithm to remove mismatched feature point pairs. RANSAC estimates a geometric transformation model (such as a homography matrix) by iteratively selecting feature point pairs, and then verifies other feature point pairs based on the geometric transformation model to exclude feature point pairs that do not conform to the geometric transformation model.

[0054] Based on the above process, the feature point pairs between the first orthographic image and each second orthographic image can be obtained. The first orthographic image and the second orthographic image The feature point pairs that match each other are The first orthographic image The pixel coordinates of the feature points in can be expressed as:

[0055] With the first orthographic image The second orthographic image with one-to-one correspondence between the feature points in The pixel coordinates of the feature points in can be expressed as:

[0056] in, Used to represent the first orthographic image middle The pixel coordinates of the feature points, ( , ) is used to represent the first orthographic image Middle The pixel coordinates of the feature points, Used to indicate the Feature points in the first orthographic image The horizontal coordinate in the image coordinate system is Used to indicate the Feature points in the first orthographic image The vertical coordinate in the image coordinate system. Used to represent the second orthographic image middle The pixel coordinates of the feature points, ( , ) is used to represent the second orthographic image Middle The pixel coordinates of the feature points, Used to indicate the feature points in the second orthographic image The horizontal coordinate in the image coordinates is Used to indicate the feature points in the second orthographic image The vertical coordinate in image coordinates.

[0057] In step S203, based on the matching feature point pairs between the first orthographic image and each of the second orthographic images, first mapping relationships between the first orthographic image and each of the second orthographic images on the image are respectively determined.

[0058] The first mapping relationship may be expressed in the following forms: transformation matrix, homography matrix, function, lookup table, point-to-point matching list, etc.

[0059] In one embodiment, a first mapping relationship between the first orthographic image and each second orthographic image on an image can be determined based on the pixel coordinates of the feature point pairs that match between the first orthographic image and each second orthographic image. In this solution, the pixel coordinates of the feature point pairs between the first orthographic image and each second orthographic image are accurate, so the mapping relationship between the first orthographic image and each second orthographic image, that is, the first mapping relationship, can be accurately and efficiently determined.

[0060] In step S204, based on the first mapping relationship corresponding to each second orthophoto image and the pixel coordinates of the image center point of each second orthophoto image, the pixel coordinates of each corresponding point in the first orthophoto image are determined, and each corresponding point corresponds one-to-one to the image center point of each second orthophoto image.

[0061] The image center point of the second orthographic image is the point at the exact center of the second orthographic image. The image center point may be a feature point. Of course, the image center point may also be a pixel point other than a feature point.

[0062] The corresponding points in the first orthographic image may be feature points. Of course, the corresponding points may also be pixel points other than feature points.

[0063] In one embodiment, w is used to represent the height of the orthographic image, and h is used to represent the width of the orthographic image. If the pixel coordinates of the center point of the image of the second orthographic image are represented in the form of a matrix, the pixel coordinates of the center point of the image can be represented as: Based on the first mapping relationship, the pixel coordinates of the center point of the image are mapped to the first orthographic image to obtain corresponding points, which are expressed as Therefore, based on the first mapping relationship and the pixel coordinates of the center point of the second orthographic image, the calculation formula for determining the pixel coordinates of the corresponding point can be expressed as: s =

[0064] Where s= + +1, Used to represent the first mapping relationship. In this example, It can be in the form of a matrix, where and is the parameter of the last row of the matrix, ( , ) is used to represent the pixel coordinates of the corresponding point in the first orthographic image, It is used to represent the horizontal coordinate of the corresponding point in the image coordinate system of the first orthographic image. Used to represent the ordinate of the corresponding point in the image coordinate system of the first orthographic image.

[0065] Since the first mapping relationship between the first orthographic image and each second orthographic image on the image has been determined in step S203, based on the first mapping relationship corresponding to each second orthographic image, the image center point on each second orthographic image can be quickly mapped to the first orthographic image, and in the first orthographic image, the corresponding point corresponding to the image center point is determined.

[0066] In step S205, a second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system is determined based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point.

[0067] The second mapping relationship may be expressed in the following forms: perspective transformation matrix, homography matrix, function, lookup table, point-to-point matching list, etc.

[0068] In the scenario where a drone inspects a photovoltaic power station, since each second orthographic image is taken with the lens of the image acquisition device carried by the drone pointed vertically downward at the ground, the geographic coordinates of the second orthographic image can be directly determined as the geographic coordinates of the image center point of the second orthographic image.

[0069] In one embodiment, the process of determining the geographic coordinates of the image center point of each second orthophoto image includes: determining the geographic coordinates of each second orthophoto image based on the latitude and longitude parameters of each second orthophoto image; and determining the geographic coordinates of each second orthophoto image as the geographic coordinates of the image center point of each second orthophoto image.

[0070] There are multiple ways to obtain the geographic coordinates of the second orthographic image.

[0071] In one embodiment, the geographic coordinates of the second orthographic image may be directly acquired through an image acquisition device.

[0072] In another embodiment, the geographic coordinates of each second orthographic image may be determined based on the latitude and longitude parameters of each second orthographic image.

[0073] There are many implementation methods for determining the geographic coordinates of each second orthographic image based on the latitude and longitude parameters of each second orthographic image.

[0074] In one embodiment, the latitude and longitude parameters of the second orthophoto images are converted into the National Geodetic Coordinate System 2000 (CGCS2000) coordinate system to obtain the geographic coordinates of each second orthophoto image.

[0075] In another embodiment, the longitude and latitude parameters of the second orthoimages are converted into the World Geodetic System 84 (WGS84) coordinate system to obtain the geographic coordinates of each second orthoimage.

[0076] In another embodiment, the longitude and latitude parameters of each second orthographic image are converted into the Beidou coordinate system to obtain the geographic coordinates of each second orthographic image.

[0077] In the scenario where the drone inspects the photovoltaic power station, since the first orthographic image is also taken by the lens of the image acquisition device configured by the drone pointing vertically downward at the ground, the geographic coordinates of the first orthographic image can be determined as the geographic coordinates of the image center point of the first orthographic image. The method for determining the geographic coordinates of the first orthographic image can refer to the method for determining the geographic coordinates of the second orthographic image described above, which will not be repeated here.

[0078] In one embodiment, a second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image may be determined based on the pixel coordinates and geographic coordinates of the image center point of the first orthographic image, and the pixel coordinates and geographic coordinates of each corresponding point in the first orthographic image.

[0079] Based on the latitude and longitude parameters of the first orthographic image and each second orthographic image, the geographic coordinates of the first orthographic image and each second orthographic image are determined, which can improve the accuracy of the geographic coordinates of the first orthographic image and each second orthographic image, thereby improving the accuracy of the second mapping relationship.

[0080] There are multiple ways to obtain the longitude and latitude parameters of each second orthographic image.

[0081] In one embodiment, the longitude and latitude parameters of each second orthographic image may be determined by manual labeling.

[0082] In another embodiment, it can be directly obtained from the Exchangeable Image File Format (EXIF) of each second orthophoto image. EXIF ​​file refers to metadata embedded in the image. The EXIF ​​file records detailed information about the shooting conditions of the image, related settings of the image acquisition device, and editing software. And EXIF ​​files can be applied to various types of image formats, such as JPEG format, PNG format, and TIFF format.

[0083] In the scenario where a drone inspects a photovoltaic power station, if the first orthographic image and the second orthographic image captured by the image acquisition device configured for the drone are in JPEG format, the corresponding longitude and latitude parameters of the first orthographic image and each second orthographic image can be directly obtained from the EXIF ​​files corresponding to the first orthographic image and each second orthographic image.

[0084] In step S206, the geographical range of the first orthographic image is determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

[0085] Wherein, according to actual needs, the geographical scope of the first orthographic image may be the geographical scope of the real environment corresponding to the first orthographic image, or may be the geographical scope of a part of the real environment corresponding to the first orthographic image.

[0086] The positioning point in the first orthographic image may be a feature point. Of course, the positioning point may also be a pixel point other than a feature point.

[0087] In actual application, the geographical range of the first orthographic image may be determined based on the pixel coordinates of different numbers of positioning points and the second mapping relationship.

[0088] In an embodiment, when the number of the positioning points is two, the two positioning points may be located on the same diagonal line of the first orthographic image.

[0089] In another embodiment, the positioning points may be at least three pixel points that are not in a straight line. Since three points that are not in a straight line can determine a plane, the geographical range of the first orthographic image can be determined relatively quickly based on the pixel coordinates of the three pixel points that are not in a straight line in the first orthographic image.

[0090] For example, when the number of the positioning points is three, any two of the three positioning points may be located on the same side of the first orthographic image, and the remaining one positioning point may be located on the side opposite to the side where the any two positioning points are located.

[0091] For example, when the number of the positioning points is four, the four positioning points may be four corner points of the first orthographic image.

[0092] In the embodiment of the present application, since the first orthographic image and each second orthographic image have an image overlap area, image matching is performed on the first orthographic image and multiple second orthographic images, and feature point pairs having a mapping relationship between the first orthographic image and each second orthographic image can be determined. By using the feature point pairs between the first orthographic image and each second orthographic image, the mapping relationship between the first orthographic image and each second orthographic image on the image, that is, the first mapping relationship, can be accurately and efficiently determined, so that by using the first mapping relationship corresponding to each second orthographic image, the image center point of the second orthographic image can be quickly mapped to the first orthographic image, and the corresponding point corresponding to the image center point is determined in the first orthographic image. Since the geographic coordinates of the image center point of each second orthographic image are known accurately, and the image center point of each second orthographic image corresponds one-to-one to the corresponding point in the first orthographic image, the second mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image can be accurately determined based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point. Since the second mapping relationship accurately describes the mapping relationship between the image space of the first orthographic image and the physical space, the geographical scope of the first orthographic image can be quickly and accurately determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship. It can be seen that in the process of determining the geographical scope of the first orthographic image, the present application cleverly utilizes the mapping relationship between multiple orthographic images and the accurate geographical coordinates of the center point of the image to quickly and accurately determine the geographical scope of the first orthographic image. On this basis, the accurately estimated geographical scope can be used to stitch and process multiple images of the photovoltaic power station, analyze the overall area of ​​the photovoltaic power station and locate problems, and effectively improve the daily management and maintenance efficiency of the photovoltaic power station.

[0093] The geometric transformation of an image is used to describe the changes in the position, size, shape, etc. of an image. It can include rigid transformation, similarity transformation, affine transformation, and perspective transformation, among which rigid transformation can involve translation and rotation between images, similarity transformation involves scaling and shearing between images, and perspective transformation is to project an image to a new viewing plane, also known as projection mapping. Perspective transformation can simultaneously handle rotation, translation, scaling, and shearing between two images, while affine transformation is a linear transformation and can be regarded as a special case of perspective transformation. Even if the two images are taken by the image acquisition device at different locations, the correct mapping relationship between the two images can be accurately found.

[0094] In an embodiment of the present application, a first transformation matrix may be used to represent the first mapping relationship, wherein the first transformation matrix includes multiple transformation parameters. The following embodiment takes the most complex perspective transformation as an example to describe the determination of the first mapping relationship between the first orthographic image and the second orthographic image, i.e., using the first perspective transformation matrix including multiple perspective transformation parameters as an example of the first transformation matrix, but it can be understood that under the overall concept of the present application, the perspective transformation can be replaced by other geometric transformations as needed.

[0095] like Figure 3 As shown, another embodiment of the present application provides a flowchart for determining a first mapping relationship based on perspective transformation. Figure 3 The difference from other embodiments is that step S203 in other embodiments can be specifically refined as follows: Figure 3 Steps S2031 and S2032 in .

[0096] In step S2031, a first perspective transformation matrix is ​​constructed, wherein the first perspective transformation matrix contains a plurality of perspective transformation parameters.

[0097] Among them, the perspective transformation parameters have properties such as rotation, translation, scaling, shearing and perspective transformation.

[0098] In one embodiment, the constructed first perspective transformation matrix H may be a 3 3. Each element in the first perspective transformation matrix H is a perspective transformation parameter. For example, the constructed first perspective transformation matrix H may contain 8 unknown perspective transformation parameters, so the first perspective transformation matrix H can be expressed as

[0099] As a non-limiting example, a, b, d and e are perspective transformation parameters for rotation, scaling and shearing, c and f are perspective transformation parameters for translation, and g and h are perspective transformation parameters for perspective transformation.

[0100] In step S2032, for the first orthographic image and any second orthographic image, based on the first perspective transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any second orthographic image, the parameter values ​​of multiple perspective transformation parameters of the first perspective transformation matrix are determined, and the first perspective transformation matrix is ​​used to characterize the first mapping relationship.

[0101] Since the first perspective transformation matrix is ​​used to characterize the mapping relationship between the corresponding first orthographic image and the second orthographic image on the image, the mapping relationship between the first orthographic image and the second orthographic image can be constructed based on the constructed first perspective transformation matrix containing unknown perspective transformation parameters and the corresponding feature point pairs between the first orthographic image and the second orthographic image.

[0102] For example, taking the first orthographic image and the second orthographic image Taking the first orthographic image as an example, the determination of and the second orthographic image The first perspective transformation matrix between The process includes a first construction step, a second construction step and a solution step, wherein: First build step: build a The first perspective transformation matrix with 8 unknown perspective transformation parameters

[0103]

[0104] in, , , , , , , as well as is the first perspective transformation matrix The perspective transformation parameters in .

[0105] Second construction step: based on the first orthographic image and the second orthographic image A feature point pair between and , construct the first orthographic image and the second orthographic image The mapping relationship between them is =

[0106] Where s= + +1, Used to represent the first orthographic image The pixel coordinates of the kth feature point in , Used to indicate the kth feature point in the first orthographic image The horizontal coordinate in the image coordinate system is Used to indicate the kth feature point in the first orthographic image The vertical coordinate in the image coordinate system. Used to represent the second orthographic image The pixel coordinates of the kth feature point in , Used to represent the kth feature point in the second orthographic image The horizontal coordinate in the image coordinate system is Used to represent the kth feature point in the second orthographic image The vertical coordinate in the image coordinate system.

[0107] Solution steps: = Calculate and you can get ( )= + +

[0108] ( )= + +

[0109] Arranged

[0110]

[0111] The matrix can be expressed as =

[0112] It can be simplified as: = , where A is used to represent , B is used to represent .

[0113] Since the constructed first perspective transformation matrix contains 8 perspective transformation parameters, in the first orthographic image and the second orthographic image After there are at least 4 feature point pairs between them, the first perspective transformation matrix can be directly solved ,Right now = , thus obtaining the first orthographic image and the second orthographic image The first perspective transformation matrix between , the first perspective transformation matrix Can be used to characterize the first orthographic image and the second orthographic image The first mapping relationship between .

[0114] Since the pixel coordinates of the feature point pairs between the first orthographic image and each second orthographic image are known and accurate, the parameter values ​​of multiple transformation parameters of the first transformation matrix can be determined by utilizing the feature point pairs between the first orthographic image and each second orthographic image. The parameter values ​​of multiple transformation parameters of the first transformation matrix can be accurately determined, so that the determined first transformation matrix can accurately reflect the mapping relationship between the first orthographic image and the corresponding second orthographic image.

[0115] In some embodiments, the second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system can be represented by a second transformation matrix, wherein the second transformation matrix includes a plurality of transformation parameters. In the following embodiments, the second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system is described by taking perspective transformation as an example, that is, a second perspective transformation matrix including a plurality of perspective transformation parameters is used as an example of the second transformation matrix, but it can be understood that, under the overall concept of the present application, the perspective transformation can be replaced by other geometric transformations as needed.

[0116] like Figure 4 As shown, another embodiment of the present application provides a flowchart for determining a second mapping relationship based on perspective transformation. Figure 4 The difference from other embodiments is that step S205 in other embodiments can be specifically refined as follows: Figure 4 Steps S2051 to S2053 in .

[0117] In step S2051, the geographical coordinates of the center point of each image are used as the geographical coordinates of the corresponding points.

[0118] Since the image center point of each second orthographic image corresponds one-to-one to each corresponding point in the first orthographic image, the geographical coordinates of the image center point of each second orthographic image can be directly determined as the geographical coordinates of each corresponding point.

[0119] In step S2052, a second perspective transformation matrix is ​​constructed, where the second perspective transformation matrix contains a plurality of perspective transformation parameters.

[0120] The process of constructing the second perspective transformation matrix is ​​the same as the process of constructing the first perspective transformation matrix in the above example, and will not be repeated here.

[0121] In step S2053, based on the second perspective transformation matrix, the geographic coordinates and the pixel coordinates of each corresponding point, parameter values ​​of a plurality of perspective transformation parameters of the second perspective transformation matrix are determined, and the second perspective transformation matrix is ​​used to represent the second mapping relationship.

[0122] The process of determining the parameter values ​​of the multiple perspective transformation parameters of the second perspective transformation matrix is ​​the same as the process of determining the parameter values ​​of the multiple perspective transformation parameters of the first perspective transformation matrix in the above example, and will not be repeated here.

[0123] Since the geographic coordinates of each second orthographic image are known and accurate, the geographic coordinates of each second orthographic image are determined as the geographic coordinates of the image center point of each second orthographic image, so that the geographic coordinates of the image center point of each second orthographic image can be accurately determined. Since each image center point corresponds to each corresponding point one by one, the geographic coordinates of each image center point are determined as the geographic coordinates of each corresponding point, so that the geographic coordinates of each corresponding point can be accurately determined. Based on the geographic coordinates and pixel coordinates of each corresponding point, the parameter values ​​of multiple transformation parameters of the second transformation matrix are determined, so that the parameter values ​​of multiple transformation parameters in the second transformation matrix can be accurately determined, so that the determined second mapping relationship can accurately reflect the mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image.

[0124] In one embodiment, parameter values ​​of a plurality of perspective transformation parameters of the second perspective transformation matrix are determined based on the second perspective transformation matrix, the geographic coordinates and pixel coordinates of the image center point of the first orthographic image, and the geographic coordinates and pixel coordinates of each corresponding point.

[0125] For example, if w is used to represent the height of the orthographic image and h is used to represent the width of the orthographic image, then the first orthographic image The pixel coordinates of the center point of the image can be expressed as . First orthographic image The pixel coordinates of the image center point and each corresponding point in can be expressed as =

[0126] in, In ( , ) is used to represent the first orthographic image The pixel coordinates of the center point of the image, In ( , ) is used to represent the first orthographic image The pixel coordinates of the jmth corresponding point in , Used to indicate the jmth corresponding point in the first orthographic image The horizontal coordinate in the image coordinate system is Used to represent jm corresponding points in the first orthographic image The vertical coordinate in the image coordinate system.

[0127] Since the geographical coordinates of the image center point of each second orthoimage are the same as the first orthoimage The geographical coordinates of the corresponding points in the first orthographic image The image center point and the geographic coordinates of each corresponding point can be expressed as

[0128] in, In ( , ) is used to represent the first orthographic image The geographical coordinates of the center point of the image, ( , ) is used to represent the first orthographic image The geographic coordinates of the jmth corresponding point in , Used to represent the horizontal coordinate of the jmth corresponding point in the geographic coordinate system. Used to represent the ordinate of the jmth corresponding point in the geographic coordinate system.

[0129] Therefore, the process of determining the parameter values ​​of the perspective transformation parameters of the first perspective transformation matrix in the above example can be referred to. and matrix , determine the parameter values ​​of the perspective transformation parameters in the second perspective transformation matrix.

[0130] In actual application, each corner point of an image can constitute the entire coordinate range of the image. Based on this, Figure 5 As shown, in yet another embodiment of the present application, a flowchart is provided for determining the geographical range of the first orthographic image based on each corner point of the first orthographic image. Figure 5 The difference from other embodiments is that step S206 in other embodiments can be specifically refined as follows: Figure 5 Step S2061 and step S2062 in .

[0131] In step S2061, the geographic coordinates of each corner point of the first orthographic image are obtained based on the pixel coordinates of each corner point of the first orthographic image and the second mapping relationship.

[0132] Each corner point of the first orthographic image may be a feature point. Of course, each corner point of the first orthographic image may also be a pixel point other than a feature point.

[0133] Each corner point of the first orthographic image may be a point where two mutually perpendicular sides of the first orthographic image intersect. Therefore, the number of corner points of the first orthographic image may be four.

[0134] In step S2062, the geographical range of the first orthographic image is determined based on the geographical coordinates of each corner point of the first orthographic image.

[0135] Since the second mapping relationship is the mapping relationship between the image coordinate system and the geographic coordinate system of the first orthographic image, the pixel coordinates of each corner point of the first orthographic image can be mapped based on the second mapping relationship to quickly and accurately obtain the geographic coordinates of each corner point, thereby quickly determining the geographic scope of the first orthographic image based on the geographic coordinates of each corner point.

[0136] For example, if w is used to represent the height of the orthographic image and h is used to represent the width of the orthographic image, then the first orthographic image The four corner points ( , , as well as ) can be expressed as: , , ,

[0137] If adopted The first orthographic image The four corner points ( , , as well as ) and the geographical coordinates of = express For the first orthographic image Corner point , then =

[0138] in, Used to indicate corner points The geographical coordinates of To indicate corner points The horizontal coordinate in the geographic coordinate system, Used to indicate corner points The vertical coordinate in the geographic coordinate system, Used to represent the second mapping relationship, s and The parameters in the last row of the matrix are related and used for normalization The homogeneous coordinate form of .

[0139] Based on the second mapping relationship , respectively, the diagonal points , and corner points Mapping is performed to obtain the corresponding , as well as , and then the first orthographic image The four geographic coordinates can be expressed as: = , = , = , =

[0140] in, Used to indicate corner points The geographical coordinates of Used to indicate corner points The horizontal coordinate in the geographic coordinate system, Used to indicate corner points The vertical coordinate in the geographic coordinate system, Used to indicate corner points The geographical coordinates of Used to indicate corner points The horizontal coordinate in the geographic coordinate system, Used to indicate corner points The vertical coordinate in the geographic coordinate system, Used to indicate corner points The geographical coordinates of Used to indicate corner points The horizontal coordinate in the geographic coordinate system, Used to indicate corner points The vertical coordinate in the geographic coordinate system.

[0141] Depend on , as well as , can form the first orthographic image geographical scope.

[0142] In addition, for the first orthographic image The other pixel points except the four corner points can be obtained by bilinear interpolation method. The geographic coordinates of all pixel points except the four corner points.

[0143] In some examples, the image processing method of the present application also includes: performing distortion correction on the first orthographic image and multiple second orthographic images to obtain the distortion-corrected first orthographic image and multiple second orthographic images, and the distortion-corrected first orthographic image and multiple second orthographic images are used to extract feature points.

[0144] By performing distortion correction on the first orthographic image and the plurality of second orthographic images, geometric distortion introduced in the imaging process can be eliminated, further ensuring that feature points of the first orthographic image and the plurality of second orthographic images after the distortion correction can be extracted more accurately subsequently.

[0145] In one embodiment, the first orthographic image and the plurality of second orthographic images may be subjected to distortion correction by the following steps, wherein the specific process includes a first determination step, a distortion correction step and a distortion correction processing step, wherein: The first determination step: determine the camera intrinsic parameter matrix K. The camera intrinsic parameter matrix K can be expressed as

[0146] in, It is used to represent the effective focal length of the first orthographic image or each second orthographic image on the x-axis, It is used to represent the effective focal length of the first orthographic image or each second orthographic image on the y-axis, and is the principal point coordinate, that is Used to represent the horizontal coordinate of the center point of the image in the pixel coordinate system. Used to represent the vertical coordinate of the center point of the image in the pixel coordinate system.

[0147] Distortion correction steps: Image distortion correction coefficients usually include radial distortion coefficients and tangential distortion coefficients. The radial distortion coefficients include , as well as , the tangential distortion coefficients include and Based on this, the distortion model of the first orthographic image or each second orthographic image can be expressed as (1+ + )+2 xy+ ) (1+ + )+ xy+ ) in,( ) is used to represent the pixel coordinates of the pixel points in the first orthographic image or each second orthographic image after distortion correction, used to represent the horizontal coordinate of a pixel point in the first orthographic image or each second orthographic image after distortion correction, It is used to represent the ordinate of the pixel point in the first orthographic image or each second orthographic image after distortion correction. (x, y) is the pixel coordinate of the pixel point in the first orthographic image or each second orthographic image before distortion correction. x is used to represent the abscissa of the pixel point in the first orthographic image or each second orthographic image before distortion correction. y is used to represent the ordinate of the pixel point in the first orthographic image or each second orthographic image before distortion correction. = .

[0148] Distortion correction processing step: construct a mapping matrix, and iteratively interpolate the pixel coordinates of the pixel points in the first orthophoto image or each second orthophoto image before distortion correction in the distortion correction step and the pixel coordinates of the corresponding pixel points after distortion correction, so as to obtain a mapping matrix between the first orthophoto image or each second orthophoto image before and after distortion correction, that is, obtain a mapping table of distorted coordinates and undistorted coordinates of the first orthophoto image or each second orthophoto image. Then, based on the mapping table, one-to-one mapping is performed on the pixel points in the first orthophoto image or each second orthophoto image to obtain the corresponding image after distortion correction.

[0149] For ease of understanding, the following uses an example of a drone inspecting a photovoltaic power station to illustrate the processing process of the image processing method of the present application. Figure 6 The flowchart of the image processing method for the scenario of the drone inspecting the photovoltaic power station provided in the embodiment of the present application includes steps S601 to S608.

[0150] In step S601, the drone inspects the photovoltaic power station and takes n orthographic images, that is, the orthographic image set .

[0151] In step S602, a first orthographic image and a plurality of second orthographic images are determined from the plurality of orthographic images.

[0152] In the estimation When the geographical scope of As the first orthographic image. and The geographical distance between them is determined as For ease of understanding, assume that and If the geographical distance of each orthographic image in satisfies the preset conditions, The orthographic images in the image are all determined as the second orthographic images.

[0153] In step S603, distortion correction is performed on the first orthographic image and the plurality of second orthographic images.

[0154] In step S604, image matching is performed on the first orthographic image and each of the second orthographic images to determine feature point pairs between the first orthographic image and each of the second orthographic images.

[0155] The SIFT algorithm is used to extract feature points of the first orthographic image and each second orthographic image. Based on the feature vector distance between the feature points of the first orthographic image and the feature points of each second orthographic image, the first orthographic image and each second orthographic image are image matched to determine the feature point pairs between the first orthographic image and each second orthographic image.

[0156] In step S605 , based on the feature point pairs between the first orthographic image and each of the second orthographic images, a first mapping relationship between the first orthographic image and each of the second orthographic images on the image is determined.

[0157] In step S606, based on the first mapping relationship, a corresponding point corresponding to the image center point of the second orthographic image is determined in the first orthographic image.

[0158] In step S607, a second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system is determined based on the pixel coordinates and geographic coordinates of the image center point of the first orthographic image and the pixel coordinates and geographic coordinates of the corresponding point.

[0159] Since the first orthographic image and each second orthographic image are taken when the lens of the image acquisition device equipped with the drone is pointed vertically downward at the ground, the geographic coordinates of the first orthographic image can be determined as the geographic coordinates of the image center point of the first orthographic image, and the geographic coordinates of each second orthographic image can be determined as the geographic coordinates of the image center point of each second orthographic image.

[0160] Since each corresponding point in the first orthographic image has a one-to-one correspondence with the image center point of each second orthographic image, the geographic coordinates of the image center point of each second orthographic image can be used as the geographic coordinates of each corresponding point in the first orthographic image.

[0161] Based on this, the geographical coordinates of the image center point and corresponding points in the first orthographic image can be obtained.

[0162] The process of determining the geographical coordinates of the image center point of the first orthographic image and the image center points of each second orthographic image is as follows: From the EXIF ​​files of the first orthographic image and each of the second orthographic images, the latitude and longitude parameters of the first orthographic image and each of the second orthographic images are determined, which can be specifically expressed as LON={ } LAT={ } The LON set is a longitude parameter set, and the LON set element It is used to represent the longitude parameter of the nth orthographic image. The LAT set is a latitude parameter set, and the LAT set contains Used to represent the latitude parameter of the nth orthographic image.

[0163] Based on the latitude and longitude parameters of the first orthographic image and each of the second orthographic images, the geographic coordinates of the first orthographic image and each of the second orthographic images are determined. Specifically, it can be expressed as: LX=

[0164] LY={ } Among them, the LX set is the horizontal axis set, and the LX set It is used to represent the horizontal coordinate of the nth orthographic image in the geographic coordinate system. The LY set is the vertical coordinate set. Used to represent the vertical coordinate of the nth orthographic image in the geographic coordinate system.

[0165] In step S608, the geographical range of the first orthographic image is determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

[0166] Based on the same inventive concept, the embodiment of the present application also provides an image processing device. Figure 7 The image processing device provided in the embodiment of the present application is described in detail.

[0167] Figure 7 It is a structural schematic diagram of an image processing device provided in an embodiment of the present application.

[0168] like Figure 7 As shown, the image processing device 700 may include an acquisition unit 710 , a matching unit 720 , a first determination unit 730 , a second determination unit 740 , a third determination unit 750 , and a fourth determination unit 760 .

[0169] The acquisition unit 710 may be used to acquire a first orthographic image and a plurality of second orthographic images, wherein the first orthographic image and each of the second orthographic images have an image overlapping area.

[0170] The matching unit 720 may be used to perform image matching on the first orthographic image and each second orthographic image, and determine matching feature point pairs between the first orthographic image and each second orthographic image.

[0171] The first determining unit 730 may be configured to determine first mapping relationships between the first orthographic image and each of the second orthographic images on the images based on the feature point pairs that match between the first orthographic image and each of the second orthographic images.

[0172] The second determination unit 740 can be used to determine the pixel coordinates of each corresponding point in the first orthographic image based on the first mapping relationship corresponding to each second orthographic image and the pixel coordinates of the image center point of each second orthographic image, and each corresponding point corresponds one-to-one to the image center point of each second orthographic image.

[0173] The third determining unit 750 may be configured to determine a second mapping relationship between the image coordinate system of the first orthographic image and the geographic coordinate system based on the pixel coordinates of each corresponding point and the geographic coordinates of each image center point.

[0174] The fourth determining unit 760 may be configured to determine the geographic range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

[0175] In one embodiment, the first determination unit can be used to construct a first transformation matrix, which contains multiple transformation parameters; for the first orthographic image and any second orthographic image, based on the first transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any second orthographic image, the parameter values ​​of the multiple transformation parameters of the first transformation matrix are determined, and the first transformation matrix is ​​used to characterize the first mapping relationship.

[0176] The third determining unit can be used to determine the geographic coordinates of each second orthophoto image based on the latitude and longitude parameters of each second orthophoto image; and determine the geographic coordinates of each second orthophoto image as the geographic coordinates of the image center point of each second orthophoto image.

[0177] The third determination unit can also be used to use the geographic coordinates of the center point of each image as the geographic coordinates of the corresponding points; construct a second transformation matrix, the second transformation matrix contains multiple transformation parameters; based on the second transformation matrix, the geographic coordinates of each corresponding point and the pixel coordinates, determine the parameter values ​​of the multiple transformation parameters of the second transformation matrix, the second transformation matrix is ​​used to characterize the second mapping relationship.

[0178] In one embodiment, the fourth determination unit can also be used to obtain the geographic coordinates of each corner point of the first orthographic image based on the pixel coordinates of each corner point of the first orthographic image and the second mapping relationship when the positioning point is a corner point of the first orthographic image; and determine the geographic range of the first orthographic image based on the geographic coordinates of each corner point of the first orthographic image.

[0179] In one embodiment, the acquisition unit may be further configured to select, from the plurality of orthographic images, a plurality of second orthographic images having image overlapping areas with the first orthographic image.

[0180] The acquisition unit can also be used to determine the geographical distance between the first orthographic image and each orthographic image based on the geographical coordinates of the first orthographic image and each orthographic image; determine whether the geographical distance meets a preset condition; and when the geographical distance meets the preset condition, determine the orthographic image as the second orthographic image.

[0181] In one embodiment, the image processing device further includes a distortion correction unit. The distortion correction unit can also be used to perform distortion correction on the first orthographic image and the plurality of second orthographic images to obtain the first orthographic image and the plurality of second orthographic images after distortion correction, and the first orthographic image and the plurality of second orthographic images after distortion correction are used to extract feature points.

[0182] It should be noted that the image processing device is a device corresponding to the above-mentioned image processing method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects, which will not be repeated here.

[0183] Figure 8 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0184] The electronic device may include a processor 801 and a memory 802 storing computer program instructions.

[0185] Specifically, the processor 801 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0186] The memory 802 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 802 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 802 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 802 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 802 is a non-volatile solid-state memory.

[0187] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0188] The processor 801 implements any one of the image processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 802 .

[0189] In one example, the electronic device may further include a communication interface 803 and a bus 810. Figure 8 As shown, the processor 801, the memory 802, and the communication interface 803 are connected via a bus 810 and communicate with each other.

[0190] The communication interface 803 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0191] Bus 810 includes hardware, software or both, coupling components to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 810 may include one or more buses. Although the present application embodiment describes and illustrates a specific bus, the present application considers any suitable bus or interconnect.

[0192] In addition, in combination with the image processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image processing methods in the above embodiment is implemented.

[0193] An embodiment of the present application also provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes an image processing method as provided in the embodiment of the present application.

[0194] An embodiment of the present application also provides an electronic device, which includes: a processor and a memory storing computer program instructions, wherein the processor is coupled to the memory; when the processor executes the computer program instructions, the image processing method provided in the embodiment of the present application is implemented.

[0195] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0196] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0197] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0198] The above describes various aspects of the present application with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0199] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image processing method, characterized in that: include: Acquire a first orthographic image and a plurality of second orthographic images, wherein the first orthographic image and each of the second orthographic images have an image overlapping area; performing image matching on the first orthographic image and each of the second orthographic images to determine matching feature point pairs between the first orthographic image and each of the second orthographic images; Based on the feature point pairs matched between the first orthographic image and each of the second orthographic images, respectively determine a first mapping relationship between the first orthographic image and each of the second orthographic images on the image; determining pixel coordinates of corresponding points in the first orthographic image based on the first mapping relationship corresponding to each of the second orthographic images and the pixel coordinates of the image center point of each of the second orthographic images, wherein each of the corresponding points corresponds one-to-one to the image center point of each of the second orthographic images; Determining a second mapping relationship between an image coordinate system and a geographic coordinate system of the first orthographic image based on the pixel coordinates of each of the corresponding points and the geographic coordinates of each of the image center points; The geographical range of the first orthographic image is determined based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

2. The method according to claim 1, characterized in that Respectively determining first mapping relationships between the first orthographic image and each of the second orthographic images on an image, comprising: Constructing a first transformation matrix, wherein the first transformation matrix contains a plurality of transformation parameters; For the first orthographic image and any one of the second orthographic images, the parameter values ​​of the multiple transformation parameters of the first transformation matrix are determined based on the first transformation matrix and the pixel coordinates of the feature point pairs between the first orthographic image and any one of the second orthographic images, and the first transformation matrix is ​​used to characterize the first mapping relationship.

3. The method according to claim 1, characterized in that The process of determining the geographic coordinates of the image center point of each second orthographic image comprises: Determining the geographic coordinates of each of the second orthographic images based on the latitude and longitude parameters of each of the second orthographic images; The geographic coordinates of each of the second orthographic images are determined as the geographic coordinates of the image center point of each of the second orthographic images.

4. The method according to claim 1, characterized in that: Determining a second mapping relationship between an image coordinate system and a geographic coordinate system of the first orthographic image based on the pixel coordinates of each of the corresponding points and the geographic coordinates of each of the image center points includes: Using the geographical coordinates of the center points of each of the images as the geographical coordinates of the corresponding points; constructing a second transformation matrix, wherein the second transformation matrix contains a plurality of transformation parameters; Based on the second transformation matrix, the geographic coordinates and the pixel coordinates of each of the corresponding points, parameter values ​​of the plurality of transformation parameters of the second transformation matrix are determined, and the second transformation matrix is ​​used to represent the second mapping relationship.

5. The method according to claim 1, characterized in that The positioning points are at least three pixel points that are not on a straight line.

6. The method according to claim 5, characterized in that In the case where the positioning point is a corner point of the first orthographic image, Determining a geographic range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship includes: Obtaining the geographic coordinates of each of the corner points of the first orthographic image based on the pixel coordinates of each of the corner points of the first orthographic image and the second mapping relationship; The geographical range of the first orthographic image is determined based on the geographical coordinates of each of the corner points of the first orthographic image.

7. The method according to any one of claims 1 to 6, characterized in that: The step of acquiring a first orthographic image and a plurality of second orthographic images comprises: A plurality of second orthographic images having image overlapping areas with the first orthographic image are screened out from the plurality of orthographic images.

8. The method according to claim 7, characterized in that The step of selecting, from the plurality of orthographic images, a plurality of second orthographic images having image overlapping areas with the first orthographic image comprises: determining a geographical distance between the first orthographic image and each of the orthographic images based on the geographical coordinates of the first orthographic image and each of the orthographic images; Determining whether the geographical distance meets a preset condition; When the geographical distance satisfies the preset condition, the orthographic image is determined as the second orthographic image.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Distortion correction is performed on the first orthographic image and the plurality of second orthographic images to obtain the first orthographic image and the plurality of second orthographic images after distortion correction, and the first orthographic image and the plurality of second orthographic images after distortion correction are used to extract the feature points.

10. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a first orthographic image and a plurality of second orthographic images, wherein the first orthographic image and each of the second orthographic images have an image overlapping area; a matching unit, configured to perform image matching on the first orthographic image and each of the second orthographic images, and determine matching feature point pairs between the first orthographic image and each of the second orthographic images; a first determining unit, configured to respectively determine a first mapping relationship between the first orthographic image and each of the second orthographic images on an image based on the feature point pairs that match between the first orthographic image and each of the second orthographic images; a second determining unit, configured to determine the pixel coordinates of each corresponding point in the first orthographic image based on the first mapping relationship corresponding to each of the second orthographic images and the pixel coordinates of the image center point of each of the second orthographic images, wherein each of the corresponding points corresponds one-to-one to the image center point of each of the second orthographic images; a third determining unit, configured to determine a second mapping relationship between an image coordinate system and a geographic coordinate system of the first orthographic image based on the pixel coordinates of each of the corresponding points and the geographic coordinates of each of the image center points; A fourth determining unit is configured to determine a geographical range of the first orthographic image based on the pixel coordinates of the positioning point of the first orthographic image and the second mapping relationship.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image processing method as described in any one of claims 1 to 9.

13. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions, wherein the processor is coupled to the memory; when the processor executes the computer program instructions, the image processing method according to any one of claims 1 to 9 is implemented.

Images