Image parameter determination method and apparatus, storage medium, and program product

Abstract

The application discloses a method and device for determining image parameters, a storage medium and a program product. The method comprises the following steps: acquiring a first orthographic image and a second orthographic image, the first orthographic image and the second orthographic image having an image overlapping area; performing image matching on the first orthographic image and the second orthographic image to determine a matching feature point pair between the first orthographic image and the second orthographic image; constructing a geographical coordinate function of a mapping geographical point of the feature point pair based on pixel coordinates of the feature point pair, and image parameters to be determined of the first orthographic image and the second orthographic image; constructing a loss function based on the geographical coordinate function of the mapping geographical point of the feature point pair; and solving the image parameters to be determined of the first orthographic image under the condition that the loss function meets a preset loss condition, so that the image parameters of the first orthographic image can be determined quickly and accurately.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method and device for determining image parameters, a storage medium and a program product. BACKGROUND

[0002] Currently, the photovoltaic power station is routinely managed and maintained based on images of the photovoltaic power station taken by a UAV during movement.

[0003] However, the images of the photovoltaic power station taken by the UAV during movement have the problem that the image parameters are not accurate, which causes certain difficulty in the routine management and maintenance of the photovoltaic power station. SUMMARY

[0004] The embodiments of the present application provide a method and device for determining image parameters, a storage medium and a program product, which can solve the problem that the image parameters corresponding to the front-view images cannot be quickly and accurately determined.

[0005] In a first aspect, the embodiments of the present application provide a method for determining image parameters, comprising: obtaining a first front-view image and a second front-view image, the first front-view image and the second front-view image having an image overlap region; performing image matching on the first front-view image and the second front-view image to determine a matching feature point pair between the first front-view image and the second front-view image; constructing a geographical coordinate function of a mapping geographical point of the feature point pair based on pixel coordinates of the feature point pair, and image parameters to be determined of the first front-view image and the second front-view image; constructing a loss function based on the geographical coordinate function of the mapping geographical point of the feature point pair; and solving the image parameters to be determined of the first front-view image under the condition that the loss function meets a preset loss condition.

[0006] In an optional implementation of the first aspect, the loss function is related to a distance between the mapping geographical point of the feature point pair, and the loss function is constructed based on the geographical coordinate function of the mapping geographical point of the feature point pair, comprising: determining a geographical coordinate deviation between the first front-view image and the second front-view image based on a first geographical coordinate of the first front-view image and a second geographical coordinate of the second front-view image; and constructing the loss function based on the geographical coordinate deviation and the geographical coordinate function of the mapping geographical point of the feature point pair.

[0007] In an optional implementation of the first aspect, the first geographical coordinate of the first front-view image and the second geographical coordinate of the second front-view image are determined based on longitude and latitude parameters of the first front-view image and the second front-view image, comprising: determining the first geographical coordinate based on the longitude and latitude parameters of the first front-view image; determining the second geographical coordinate based on the longitude and latitude parameters of the second front-view image; and determining the geographical coordinate deviation between the first front-view image and the second front-view image based on the first geographical coordinate and the second geographical coordinate.

[0008] In an optional implementation of the first aspect, the image parameters to be determined include at least one of a yaw angle, a pitch angle, a roll angle, and an altitude.

[0009] In an optional implementation of the first aspect, constructing the geographical coordinate function of the mapped geographical point of the feature point pair based on the pixel coordinates of the feature point pair, the image parameters to be determined of the first front-view image and the second front-view image includes: constructing a first three-dimensional rotation matrix based on the yaw angle, the pitch angle, and the roll angle of the first front-view image; constructing a first scaling matrix based on the altitude of the first front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle, and the altitude is the image parameter to be determined; obtaining an intrinsic matrix of an image acquisition device, the image acquisition device being an image acquisition device that captures the first front-view image and the second front-view image; and constructing the geographical coordinate function of the mapped geographical point of the first feature point in the feature point pair based on the pixel coordinates of the first feature point, the intrinsic matrix, the first three-dimensional rotation matrix, and the first scaling matrix.

[0010] In an optional implementation of the first aspect, constructing the geographical coordinate function of the mapped geographical point of the feature point pair based on the pixel coordinates of the feature point pair, the image parameters to be determined of the first front-view image and the second front-view image includes: constructing a second three-dimensional rotation matrix based on the yaw angle, the pitch angle, and the roll angle of the second front-view image; constructing a second scaling matrix based on the altitude of the second front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle, and the altitude is the image parameter to be determined; obtaining an intrinsic matrix of an image acquisition device, the image acquisition device being an image acquisition device that captures the first front-view image and the second front-view image; and constructing the geographical coordinate function of the mapped geographical point of the second feature point in the feature point pair based on the intrinsic matrix, the pixel coordinates of the second feature point, the second three-dimensional rotation matrix, and the second scaling matrix.

[0011] In an optional implementation of the first aspect, constructing the loss function based on the geographical coordinate function of the mapped geographical point of the feature point pair includes: constructing a sub-loss function corresponding to the feature point pair based on the geographical coordinate function of the mapped geographical point of the feature point pair between the first front-view image and any second front-view image, the sub-loss function being related to a distance between the mapped geographical point of the feature point pair; and constructing the loss function based on the sub-loss functions of the feature point pairs between the first front-view image and the plurality of second front-view images.

[0012] In an optional implementation of the first aspect, constructing the loss function based on the sub-loss functions of the feature point pairs between the first front-view image and the plurality of second front-view images includes: determining a total number of the feature point pairs between the first front-view image and the plurality of second front-view images; and constructing the loss function based on the sub-loss functions of the feature point pairs between the first front-view image and the plurality of second front-view images and the total number of the feature point pairs.

[0013] In an optional implementation of the first aspect, the obtaining the first orthographic image and the second orthographic image comprises: determining, from the plurality of orthographic images, the second orthographic image that has an image overlapping area with the first orthographic image.

[0014] In an optional implementation of the first aspect, the determining, from the plurality of orthographic images, the second orthographic image that has an image overlapping area with the first orthographic image comprises: determining a geographic distance between the first orthographic image and each of the orthographic images based on geographic coordinates of the first orthographic image and the orthographic images; determining whether the geographic distance meets a preset condition; and determining the orthographic image as the second orthographic image when the geographic distance meets the preset condition.

[0015] In an optional implementation of the first aspect, before the image matching of the first orthographic image and the second orthographic image and the determining of the matched feature point pairs between the first orthographic image and the second orthographic image, the method further comprises: performing distortion correction on the first orthographic image and the second orthographic image to obtain the first orthographic image and the second orthographic image after distortion correction, and the first orthographic image and the second orthographic image after distortion correction are used for extracting the feature points.

[0016] In the second aspect, the embodiments of the present application provide a device for determining image parameters, comprising: an obtaining unit configured to obtain a first orthographic image and a second orthographic image, the first orthographic image and the second orthographic image having an image overlapping area; an image matching unit configured to perform image matching on the first orthographic image and the second orthographic image to determine matched feature point pairs between the first orthographic image and the second orthographic image; a first constructing unit configured to construct a geographic coordinate function of a mapping geographic point of the feature point pairs based on pixel coordinates of the feature point pairs and image parameters to be determined of the first orthographic image and the second orthographic image; a second constructing unit configured to construct a loss function based on the geographic coordinate function of the mapping geographic point of the feature point pairs; and a calculating unit configured to solve the image parameters to be determined of the first orthographic image when the loss function meets a preset loss condition.

[0017] In the third aspect, the embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the method for determining image parameters in the first aspect.

[0018] In the fourth aspect, the embodiments of the present application provide a computer program product, and instructions in the computer program product are executed by a processor of an electronic device to enable the electronic device to perform the method for determining image parameters in the first aspect.

[0019] In an embodiment of the present application, since the first orthophoto image and the second orthophoto image have an image overlapping area, image matching is performed on the first orthophoto image and the second orthophoto image, and the feature point pairs that match between the first orthophoto image and the second orthophoto image can be quickly determined. By using the pixel coordinates of the feature point pairs that match between the first orthophoto image and the second orthophoto image, and the image parameters to be determined of the first orthophoto image and the second orthophoto image, a geographic coordinate function of the mapped geographic points corresponding to the feature point pairs in the geographic coordinate system can be cleverly constructed, so that the geographic coordinate function contains the image parameters to be determined. Since the feature point pairs between the two orthophoto images are determined by image matching, the geographic coordinates of the mapped geographic points corresponding to the feature point pairs are also matched. Based on this, a loss function can be constructed based on the geographic coordinate function of the mapped geographic points corresponding to the feature point pairs. By solving the loss function, the image parameters to be determined that meet the preset loss conditions are obtained. It can be seen that in the method for determining image parameters of the present application, the geographic coordinate function of the mapped geographic point of the feature point pair is used to construct a loss function, so that the loss function corresponds to the geographic coordinate function of the mapped geographic point, which contains the image parameters to be determined, thereby solving the loss function and obtaining the image parameters to be determined of the first orthographic image that meets the preset loss conditions, which can quickly and accurately determine the image parameters. On this basis, when the image acquisition device configured by the drone takes multiple images during movement, it keeps the lens vertically downward facing the ground to take multiple orthographic images. According to this method, the image parameters to be determined of these orthographic images can be quickly and accurately determined, and the determined image parameters are used to splice and process the multiple images of the photovoltaic power station, thereby analyzing the overall area of ​​the photovoltaic power station and locating problems, effectively improving the daily management and maintenance efficiency of the photovoltaic power station. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 This is a schematic diagram of the architecture of a system for determining image parameters provided by one embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for determining image parameters provided by one embodiment of the present application;

[0023] Figure 3 This is a flowchart of a geographic coordinate function for constructing a mapping geographic point of a first feature point provided by an embodiment of the present application;

[0024] Figure 4is a flowchart of a function of constructing a geographical coordinate of a mapping geographical point of a second feature point provided by an embodiment of the present application;

[0025] Figure 5 is a flowchart of constructing a loss function of a function of a geographical coordinate of a mapping geographical point based on a feature point pair provided by an embodiment of the present application;

[0026] Figure 6 is a flowchart of a method of determining an image parameter in a scene of a UAV performing inspection on a photovoltaic power station provided by an embodiment of the present application;

[0027] Figure 7 is a structural schematic diagram of a device of determining an image parameter provided by another embodiment of the present application;

[0028] Figure 8 is a structural schematic diagram of an electronic device provided by another embodiment of the present application.

[0029] Among the above-mentioned drawings, the following reference signs are included:

[0030] 100, electronic device; 200, UAV; 700, device of determining an image parameter; 710, acquisition unit; 720, image matching unit; 730, first construction unit; 740, second construction unit; 750, calculation unit; 801, processor; 802, memory; 803, communication interface; 810, bus. DETAILED DESCRIPTION

[0031] The features and exemplary embodiments of various aspects of the present application will be described in detail below, in order to make the purposes, technical solutions and advantages of the present application more clear and apparent, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, but not to limit the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by showing examples of the present application.

[0032] It is to be understood that the terminology used herein such as first and second, and the like, is only used to distinguish one from another among two or more entities or operations, and does not necessarily require or imply that there is any such actual relationship or order between or among these entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by an "comprising" statement is not meant to exclude the existence of additional elements of the process, method, article, or apparatus that includes the stated elements.

[0033] At present, the photovoltaic power station is generally managed and maintained daily based on the photographed image of the photovoltaic power station. However, there are some problems of inaccurate related image parameters in the photographed image of the photovoltaic power station, which leads to the inability to further process the image, such as splicing multiple images of the photovoltaic power station photographed into a continuous image, so as to analyze and locate the problem of the overall area of the photovoltaic power station, and further leads to the low efficiency of the daily management and maintenance of the photovoltaic power station. For example, in the scene of using a UAV to patrol the photovoltaic power station, the image acquisition device configured by the UAV can photograph the image of the photovoltaic power station on the patrol route. However, due to the inaccurate related image parameters of the photovoltaic power station image photographed by the UAV, it leads to the inability to further process the image, such as splicing multiple images of the photovoltaic power station photographed into a continuous image, so as to accurately analyze and locate the problem of the overall area of the photovoltaic power station.

[0034] The application provides a method, device, storage medium and program product for determining image parameters. Since the first and second orthographic images have an image overlapping area, the image matching is performed on the first and second orthographic images, and the matched feature point pairs between the first and second orthographic images can be quickly determined. The pixel coordinates of the matched feature point pairs between the first and second orthographic images and the image parameters to be determined of the first and second orthographic images are used to construct a geographical coordinate function of the corresponding mapping geographical points of the feature point pairs in a geographical coordinate system, so that the image parameters to be determined are included in the geographical coordinate function. Since the feature point pairs between the two orthographic images are determined through image matching, the geographical coordinates of the corresponding mapping geographical points of the feature point pairs are also matched. Based on this, a loss function can be constructed based on the geographical coordinate function of the corresponding mapping geographical points of the feature point pairs, and the image parameters to be determined of the first orthographic image under the preset loss condition can be obtained by solving the loss function. It can be seen that, in the method for determining image parameters provided by the application, the loss function is constructed by using the geographical coordinate function of the mapping geographical points of the feature point pairs, and the loss function corresponds to the geographical coordinate function of the mapping geographical points, which includes the image parameters to be determined. Therefore, the image parameters to be determined of the first orthographic image under the preset loss condition can be obtained by solving the loss function, and the image parameters can be quickly and accurately determined. On this basis, the determined image parameters can be used to further process the images, such as stitching multiple images of a photovoltaic power station, so as to analyze and locate the problems of the whole area of the photovoltaic power station, and effectively improve the daily management and maintenance efficiency of the photovoltaic power station.

[0035] For the convenience of understanding, the application architecture of the method, device, storage medium and program product for determining image parameters provided by the application is briefly introduced by taking the unmanned aerial vehicle for inspecting the photovoltaic power station as an example. Figure 1 The application architecture of the image parameter determination system provided by the embodiment of the application is shown in the figure. The image parameter determination system comprises an unmanned aerial vehicle 200 and an electronic device 100. The unmanned aerial vehicle 200 inspects the photovoltaic power station based on an inspection route and uses the image acquisition device configured on the unmanned aerial vehicle to shoot multiple images. It should be noted that the image acquisition device configured on the unmanned aerial vehicle maintains the attitude of the lens being vertically downward to the ground when shooting multiple images, so as to ensure that the multiple images shot are orthographic images. After receiving the multiple orthographic images shot by the unmanned aerial vehicle 200, the electronic device 100 determines the image parameters of the multiple orthographic images according to the method for determining image parameters provided by the application, and obtains the determined image parameters corresponding to each orthographic image.

[0036] In actual applications, in the embodiments of the present application, the method for determining the image parameters provided in the embodiments of the present application can be implemented when the processor of the electronic device executes a program or instruction. However, in some embodiments, the image acquisition device of the drone or other devices may also have similar functions. For example, the method for determining the image parameters provided in the embodiments of the present application is implemented when the image acquisition device of the drone executes a program or instruction. The embodiments of the present application do not limit this.

[0037] It should be noted that the application scenarios described in the above embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Persons skilled in the art will appreciate that as new application scenarios emerge, the technical solutions provided by the embodiments of the present application will also be applicable to similar technical problems. The image parameter determination method provided in the embodiments of the present application can be applied to various application scenarios requiring the determination of image parameters.

[0038] The following describes the image parameter determination method, device, storage medium, and program product provided by this application.

[0039] Figure 2 FIG. 1 is a flow chart showing a method for determining image parameters provided by an embodiment of the present application. Figure 2 As shown, the method for determining image parameters provided in the embodiment of the present application may include steps S201 to S205.

[0040] In step S201 , a first orthographic image and a second orthographic image are acquired, where the first orthographic image and the second orthographic image have an image overlap area.

[0041] The first orthographic image and the second orthographic image are taken when the lens of the image acquisition device of the drone is pointed vertically downward at the ground.

[0042] The image overlap region refers to a region in the first orthographic image and the second orthographic image that has the same or similar content. Optionally, the image overlap between the first orthographic image and the second orthographic image may be complete overlap or partial overlap.

[0043] In some embodiments, any one of the plurality of orthographic images may be determined as a first orthographic image, and then a second orthographic image having an image overlap area with the first orthographic image may be determined from orthographic images other than the first orthographic image.

[0044] Taking the inspection of photovoltaic power stations by drones as an example, in actual applications, when drones perform inspection tasks, the image acquisition equipment configured by the drones takes pictures of the photovoltaic power station and obtains an orthographic image set containing n orthographic images. For example, in determining The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image. The image parameters of the first forward-looking image can be determined based on the image parameters of the at least one second forward-looking image.

[0045] Since there is an image overlap area between the first forward-looking image and the second forward-looking image, it is ensured that there are more common features between the first forward-looking image and the second forward-looking image, so that the feature point pair can be quickly and accurately determined between the first forward-looking image and the second forward-looking image.

[0046] There are various implementation manners for determining the at least one second forward-looking image having an image overlap area with the first forward-looking image from the plurality of forward-looking images.

[0047] In an embodiment, a neural network model can be used to perform target detection on the first forward-looking image and the plurality of forward-looking images, and then the intersection over union (IOU) value of the rectangular box (Bounding Box) obtained by target detection in the first forward-looking image and the rectangular box (Bounding Box) obtained by target detection in each forward-looking image is determined, and finally, based on the IOU value of the first forward-looking image and each forward-looking image, the forward-looking image corresponding to the IOU value greater than a preset threshold is determined as the second forward-looking image having an image overlap area with the first forward-looking image.

[0048] In another embodiment, the second forward-looking image having an image overlap area with the first forward-looking image can be determined based on a continuous shooting time screening manner, for example, the forward-looking image having a shooting time interval less than a preset threshold from the first forward-looking image is determined as the second forward-looking image.

[0049] In yet another embodiment, the second forward-looking image having an image overlap area with the first forward-looking image can be determined based on the geographic distance between the first forward-looking image and each forward-looking image. Specifically, the geographic distance between the first forward-looking image and each forward-looking image is determined based on the geographic coordinates of the first forward-looking image and each forward-looking image; it is determined whether the geographic distance satisfies a preset condition; and in the case where the geographic distance satisfies the preset condition, the forward-looking image is determined as the second forward-looking image.

[0050] Since the geographic coordinates of the first forward-looking image and each forward-looking image are two-dimensional plane coordinates, the geographic distance between the first forward-looking image and each forward-looking image can be determined based on the Euclidean distance between the geographic coordinates of the first forward-looking image and each forward-looking image. determining a geographical distance between the first orthographic image and each orthographic image, wherein the geographical distance is determined based on the geographical coordinates of the first orthographic image and the geographical coordinates of each orthographic image, a geographical coordinate for representing the first orthographic image, a horizontal coordinate for representing the first orthographic image in a geographical coordinate system, a vertical coordinate for representing the first orthographic image in a geographical coordinate system, a geographical coordinate for representing the orthographic image, a horizontal coordinate for representing the orthographic image in a geographical coordinate system, a vertical coordinate for representing the orthographic image in a geographical coordinate system.

[0051] The preset condition can be that the geographical distance between the first orthographic image and each orthographic image is less than a distance threshold. The distance threshold can be determined based on a shooting height, a shooting angle, and a focal length of the image acquisition device.

[0052] The orthographic images are screened based on the geographical distance between the first orthographic image and each orthographic image, so that the second orthographic image having an overlapping area with the first orthographic image can be quickly determined.

[0053] In step S202, the first orthographic image and the second orthographic image are subjected to image matching to determine a matching feature point pair between the first orthographic image and the second orthographic image.

[0054] The image matching is to determine feature points of similar parts between different images and establish a corresponding relationship between the feature points of the similar parts between the different images. The image matching can be performed on two images based on feature point matching, region matching, or global matching.

[0055] In an embodiment, the image matching is further described by taking an unmanned aerial vehicle for inspecting a photovoltaic power station as an example. First, taking a first orthographic image and a second orthographic image as an example, feature points in the first orthographic image and feature points in the second orthographic image are extracted. Then, based on a feature vector distance between the feature points in the first orthographic image and the feature points in the second orthographic image, it is determined whether the feature points in the first orthographic image and the feature points in the second orthographic image correspond to the same feature. If the feature vector distance between the feature points in the first orthographic image and the feature points in the second orthographic image meets the requirement, i.e., the two feature points correspond to the same feature, the feature points in the first orthographic image and the feature points in the second orthographic image have a corresponding relationship, so that the feature points in the first orthographic image and the feature points in the second orthographic image are determined as a feature point pair.

[0056] The feature point pair is two feature points having a corresponding relationship in two images.​​

[0057] In actual application, any feasible feature extraction method in the prior art can be adopted to extract the feature points in the first front-view image or the second front-view image. For example, the Scale Invariant Feature Transform (SIFT), Oriented FAST and Rotated BRIEF (ORB), or Binary Robust Invariant Scalable Keypoints (BRISK) algorithm, etc. can be adopted to extract the feature points in the first front-view image or the second front-view image.

[0058] In an optional embodiment, in the case where there are multiple second front-view images, the process of determining the feature point pairs between the first front-view image and each second front-view image is further described by taking the SIFT algorithm as an example. The process includes a feature extraction step, a feature matching step, a feature matching result screening step, and a false matching removal step.

[0059] Feature extraction step: SIFT feature points of the first front-view image and each second front-view image are extracted to obtain a feature point set SIFT = {SIFT1, SIFT2, …, SIFTN}, where each element in the feature point set SIFT represents a SIFT feature point in the first front-view image or the second front-view image, which is in the form of a feature vector for example but not limited thereto.

[0060] Feature matching step: the nearest neighbor matching method is adopted to determine the two feature points in each second front-view image that have the closest feature vector distance to the feature point in the first front-view image.

[0061] Feature matching result screening step: for ease of understanding, one feature point in the first front-view image and the two feature points in the corresponding second front-view image that have the closest feature vector distance are described. The feature vector distance between the feature point in the first front-view image and the two feature points in the corresponding second front-view image is screened by using a set feature threshold ​​​​​At the same time, determine the ratio of the feature vector distance between the feature point in the first orthographic image and the nearest feature point in the second orthographic image to the feature vector distance between the feature point in the first orthographic image and the next nearest feature point in the second orthographic image. In the case of , the feature point in the first orthographic image and the closest feature point in the second orthographic image are determined as a feature point pair. For example, the ratio threshold Can be set to 0.75.

[0062] Mismatch removal: Use the Random Sample Consensus (RANSAC) algorithm to remove incorrectly matched feature point pairs. RANSAC iteratively selects feature point pairs to estimate a geometric transformation model (such as a homography matrix). It then verifies the remaining feature point pairs against this geometric transformation model, eliminating those that do not conform to the geometric transformation model.

[0063] Based on the above process, the feature point pairs between the first orthographic image and each second orthographic image can be obtained. Represents the first orthographic image and the second orthographic image The feature point pairs that match between First positive image The pixel coordinates of the feature points in can be expressed as:

[0064]

[0065] With the first orthographic image The second orthographic image with one-to-one correspondence between the feature points in The pixel coordinates of the feature points in can be expressed as:

[0066]

[0067] in, Used to represent the first orthographic image middle The pixel coordinates of the feature points, ( , ) is used to represent the first orthographic image Middle The pixel coordinates of the feature points, Used to indicate the Feature points in the first orthographic image The horizontal coordinate in the image coordinate system is Used to indicate the Feature points in the first orthographic image The vertical coordinate in the image coordinate system is Used to represent the second orthographic image middle The pixel coordinates of the feature points, ( , ) is used to represent the second orthographic image Middle The pixel coordinates of the feature points, Used to indicate the Feature points in the second orthographic image The horizontal coordinate in the image coordinates, Used to indicate the Feature points in the second orthographic image The vertical coordinate in image coordinates.

[0068] In step S203 , a geographic coordinate function of the mapped geographic points of the feature point pair is constructed based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first orthographic image and the second orthographic image.

[0069] The mapped geographic point of the feature point pair is to map the feature point in the feature point pair from the image coordinate system to the geographic coordinate system to obtain the mapped geographic point corresponding to the feature point in the feature point pair.

[0070] Based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first and second orthographic images, a geographic coordinate function of the mapped geographic point of the feature point pair is constructed. That is, the image parameters to be determined of the first and second orthographic images are considered unknown parameters, and the feature points in the feature point pair are mapped from the image coordinate system to the geographic coordinate system using a camera imaging model. Thus, based on the pixel coordinates of the feature point pair, a geographic coordinate function of the mapped geographic point of the feature point pair is constructed, so that the geographic coordinate function includes the image parameters to be determined. Therefore, the geographic coordinate function of the mapped geographic point of the feature point pair is used to represent the geographic coordinates of the mapped geographic point corresponding to the feature point in the feature point pair in the geographic coordinate system.

[0071] For orthographic images, the image parameters that need to be determined may include yaw angle and / or elevation.

[0072] In one embodiment, a geographic coordinate function of the mapped geographic point of the feature point pair is constructed based on the pixel coordinates of the feature point pair and the yaw angles of the first orthographic image and the second orthographic image. Optionally, in this embodiment, the elevation can be a known accurate quantity.

[0073] In another embodiment, a geographic coordinate function of the mapped geographic point of the feature point pair is constructed based on the pixel coordinates of the feature point pair and the elevations of the first orthographic image and the second orthographic image. Optionally, in this embodiment, the yaw angle can be a known accurate quantity.

[0074] In yet another embodiment, a function of geographical coordinates of the mapping geographical points of the feature point pairs is constructed based on the pixel coordinates of the feature point pairs, the yaw angle and the altitude of the first and second orthographic images.

[0075] For the orthographic images, the angle values of the pitch angle and the roll angle are theoretically zero, but considering various errors, the angle values of the pitch angle and the roll angle can not be zero in practice. Therefore, in addition to the yaw angle and the altitude, the pitch angle and / or the roll angle can also be determined.

[0076] In summary, the embodiments of the present application do not specifically limit the image parameters that need to be determined, which can be at least one of the yaw angle, the altitude, the pitch angle, and the roll angle, or any suitable parameter that can be used to construct the geographical coordinates corresponding to the image coordinates, which can be set according to actual needs.

[0077] In step S204, a loss function is constructed based on the function of geographical coordinates of the mapping geographical points of the feature point pairs.

[0078] Since the feature point pairs between the first and second orthographic images are determined through image matching, the geographical coordinates of the mapping geographical points corresponding to the feature point pairs are also matched. Based on this, the function of geographical coordinates of the mapping geographical points of the feature point pairs is used to construct the loss function, so that the loss function corresponds to the function of geographical coordinates of the mapping geographical points. The loss function can be used to reflect the matching loss between the mapping geographical points of the feature point pairs, which can be represented by the geographical distance between the mapping geographical points of the feature point pairs, i.e., the loss function is related to the distance between the mapping geographical points of the feature point pairs. Subsequently, the image parameters of the first orthographic image can be solved when the loss function satisfies a preset loss condition, so that the obtained image parameters of the first orthographic image are more accurate.

[0079] For example, a feature point pair is used to represent a common feature between the first and second orthographic images. In the real world, the feature should correspond to a geographical coordinate. Based on this, the feature points in the feature point pair are mapped from the image coordinate system to the geographical coordinate system, and the geographical coordinates of the feature points in the feature point pair in the geographical coordinate system can be obtained. The geographical coordinates of the two feature points in the feature point pair are theoretically the same or similar, i.e., the geographical coordinates of the mapping geographical points of the feature point pair are theoretically the same or similar. Therefore, the loss function can be constructed based on the function of geographical coordinates of the mapping geographical points of the feature point pairs.

[0080] The loss function is constructed based on the geographical coordinate function of the mapping geographical points of the feature point pairs. The specific implementation manner can be: determining a geographical coordinate deviation between the first positive camera image and the second positive camera image based on the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image; and constructing the loss function based on the geographical coordinate deviation and the geographical coordinate function of the mapping geographical points of the feature point pairs.

[0081] The geographical coordinate deviation is a deviation between the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image, for example, can be a deviation between coordinate components corresponding to the first geographical coordinate and the second geographical coordinate.

[0082] There are various implementation manners for obtaining the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image. For example, the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image can be directly obtained based on an image acquisition device, or the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image can be obtained based on longitude and latitude parameters of the first positive camera image and the second positive camera image.

[0083] In some embodiments, the geographical coordinates obtained by using the pixel coordinates, the image parameters and the camera imaging model are geographical coordinates in a coordinate system of the image acquisition device itself, rather than geographical coordinates in a geodetic coordinate system. Since the shooting positions of the first positive camera image and the second positive camera image are different, this leads to a deviation between the geographical coordinates of the first positive camera image and the second positive camera image in the coordinate system of the image acquisition device itself, and accordingly, the geographical coordinates of the mapping geographical points of the feature point pairs also have a deviation. Therefore, in the process of constructing the loss function, the geographical coordinate deviation between the first positive camera image and the second positive camera image needs to be considered, so that the constructed loss function can accurately reflect the matching relationship between the geographical coordinates of the mapping geographical points of the feature point pairs matched by the first positive camera image and the second positive camera image.

[0084] In some embodiments, if the geographical coordinates obtained by using the pixel coordinates, the image parameters and the camera imaging model are geographical coordinates in a geodetic coordinate system, then in the process of constructing the loss function, the geographical coordinate deviation between the first positive camera image and the second positive camera image does not need to be considered.

[0085] There are various implementation manners for determining the geographical coordinate deviation between the first positive camera image and the second positive camera image based on the first geographical coordinate of the first positive camera image and the second geographical coordinate of the second positive camera image.

[0086] In one embodiment, the geographic coordinate deviation between the first forward-looking image and the second forward-looking image is determined based on a difference between a first geographic coordinate of the first forward-looking image and a second geographic coordinate of the second forward-looking image. As an example, if the first geographic coordinate of the first forward-looking image and the second geographic coordinate of the second forward-looking image are geographic coordinates in a coordinate system of the image acquisition device itself, the deviation between the first geographic coordinate and the second geographic coordinate can be determined directly. If the first geographic coordinate of the first forward-looking image and the second geographic coordinate of the second forward-looking image are geographic coordinates in a geodetic coordinate system, the deviation between the first geographic coordinate and the second geographic coordinate in the geodetic coordinate system can be taken as the deviation between the first geographic coordinate and the second geographic coordinate in the coordinate system of the image acquisition device itself.

[0087] In another embodiment, a first geographic coordinate is determined based on the latitude and longitude parameters of the first forward-looking image, the geographic coordinate being a geographic coordinate in a geodetic coordinate system; a second geographic coordinate is determined based on the latitude and longitude parameters of the second forward-looking image, the geographic coordinate being a geographic coordinate in a geodetic coordinate system; and a geographic coordinate deviation between the first forward-looking image and the second forward-looking image is determined based on the first geographic coordinate and the second geographic coordinate, that is, the deviation between the first geographic coordinate and the second geographic coordinate in the geodetic coordinate system is taken as the deviation between the first geographic coordinate and the second geographic coordinate in the coordinate system of the image acquisition device itself.

[0088] There are various ways to obtain the latitude and longitude parameters of the first forward-looking image and the second forward-looking image.

[0089] In one embodiment, the latitude and longitude parameters of the first forward-looking image and the second forward-looking image can be obtained by manual annotation.

[0090] In another embodiment, the latitude and longitude parameters of the first forward-looking image and the second forward-looking image can be obtained directly from an Exchangeable Image File Format (EXIF) file of the first forward-looking image and the second forward-looking image. The EXIF file refers to metadata embedded in an image. The EXIF file records the shooting conditions of the image, the related settings of the image acquisition device, and the detailed information of the editing software. The EXIF file can be applied to various types of image formats, such as JPEG format, PNG format, and TIFF format.

[0091] In the scenario of unmanned aerial vehicle (UAV) inspection of a photovoltaic power station, if the forward-looking image acquired by the image acquisition device configured on the UAV is in JPEG format, the latitude and longitude parameters corresponding to the first forward-looking image and the second forward-looking image can be obtained directly from the EXIF files corresponding to the first forward-looking image and the second forward-looking image.

[0092] There are many ways to determine the first geographic coordinates of the first orthographic image and the second geographic coordinates of the second orthographic image based on the latitude and longitude parameters of the first orthographic image and the second orthographic image.

[0093] In one embodiment, the latitude and longitude parameters of the first orthographic image and the second orthographic image are respectively converted to the National Geodetic Coordinate System 2000 (CGCS2000) coordinate system to obtain the first geographic coordinates of the first orthographic image and the second geographic coordinates of the second orthographic image.

[0094] For example, the latitude and longitude parameters of the orthographic image are determined from the EXIF ​​file of the orthographic image. The latitude and longitude parameters of the orthographic image can be expressed as

[0095] LON={ }

[0096] LAT={ }

[0097] Among them, the LON set is a longitude parameter set, and the LON set elements are It is used to represent the longitude parameter of the nth orthographic image. The LAT set is the latitude parameter set, and the LAT set contains Used to represent the latitude parameter of the nth orthographic image.

[0098] Based on the conversion of the latitude and longitude parameters of the orthographic image into the CGCS2000 coordinate system, the geographic coordinates of the orthographic image are obtained. The geographic coordinates of the orthographic image can be expressed as:

[0099] LX=

[0100] LY={ }

[0101] Among them, the LX set is the horizontal coordinate set, and the LX set It is used to represent the horizontal coordinate of the nth orthographic image in the geographic coordinate system. The LY set is the vertical coordinate set. Used to represent the vertical coordinate of the nth orthographic image in the geographic coordinate system.

[0102] In another embodiment, the latitude and longitude parameters of the first orthographic image and the second orthographic image are converted into the World Geodetic System (WGS84) coordinate system to obtain the first geographic coordinates of the first orthographic image and the second geographic coordinates of the second orthographic image.

[0103] In another embodiment, the longitude and latitude parameters of the first orthographic image and the second orthographic image are converted into the Beidou coordinate system to obtain the first geographic coordinates of the first orthographic image and the second geographic coordinates of the second orthographic image.

[0104] The feature point pair includes a first feature point and a second feature point. The loss function is constructed based on the geographic coordinate deviation and the geographic coordinate function of the mapping geographic point of the feature point pair. There are various implementation manners.

[0105] In an embodiment, the geographic coordinate function of the mapping geographic point of the first feature point is constructed based on the geographic coordinate deviation, and the loss function is constructed based on the geographic coordinate function of the mapping geographic point of the first feature point and the geographic coordinate function of the mapping geographic point of the second feature point.

[0106] In another embodiment, the geographic coordinate function of the mapping geographic point of the second feature point is constructed based on the geographic coordinate deviation, and the loss function is constructed based on the geographic coordinate function of the mapping geographic point of the second feature point and the geographic coordinate function of the mapping geographic point of the first feature point.

[0107] In yet another embodiment, the loss function is constructed based on the geographic coordinate function of the mapping geographic point of the feature point pair, and the loss function constructed based on the geographic coordinate deviation is obtained by constructing the loss function based on the geographic coordinate deviation.

[0108] The first geographic coordinate is determined based on the latitude and longitude parameters of the first front-view image, and the second geographic coordinate is determined based on the latitude and longitude parameters of the second front-view image, so that the first geographic coordinate and the second geographic coordinate can be accurately obtained. The geographic coordinate deviation between the first front-view image and the second front-view image is determined based on the first geographic coordinate and the second geographic coordinate, so that the determined geographic coordinate deviation is accurate, and the loss function can be accurately constructed based on the geographic coordinate deviation and the geographic coordinate function of the mapping geographic point of the feature point pair.

[0109] In step S205, the image parameter to be determined of the first front-view image is solved under the condition that the loss function meets the preset loss condition.

[0110] The preset loss condition can be set in various ways. For example, the preset loss condition can be that the loss function reaches a convergence state, the preset loss condition can also be that the function value of the loss function has reached a minimum, of course, the preset loss condition can also be that the number of iterations reaches a preset number of iterations, and the preset loss condition can also be that the function value of the loss function reaches a preset function value.

[0111] In an embodiment, if the geographic coordinate function of the mapping geographic point of the feature point pair is constructed based on the pixel coordinates of the feature point pair and the yaw angles of the first front-view image and the second front-view image, the yaw angle of the first front-view image can be solved when the loss function meets the preset loss condition.

[0112] In another embodiment, if a geographic coordinate function of the mapping geographic points of the feature point pair is constructed based on the pixel coordinates of the feature point pair and the elevations of the first orthographic image and the second orthographic image, the elevation of the first orthographic image can be solved when the loss function meets the preset loss condition.

[0113] In another embodiment, if a geographic coordinate function of the geographic point mapped to the feature point pair is constructed based on the pixel coordinates of the feature point pair, the yaw angle and elevation of the first orthographic image, and the yaw angle and elevation of the second orthographic image, the yaw angle and elevation of the first orthographic image can be calculated when the loss function satisfies a preset loss condition. As described above, the pitch angle, roll angle, or other image parameters of the first orthographic image can also be calculated when the loss function satisfies the preset loss condition.

[0114] There are multiple ways to solve the image parameters of the first orthographic image so that the loss function satisfies the preset loss condition.

[0115] In one embodiment, a nonlinear optimization method is used to solve the loss function until the function value of the loss function meets a preset loss condition, thereby obtaining image parameters of the first orthographic image.

[0116] When determining the image parameters of the first orthographic image using a nonlinear optimization method, specifically, a Powell optimization method may be used to solve a loss function until a function value of the loss function satisfies a preset loss condition, thereby obtaining the image parameters of the first orthographic image.

[0117] In the process of solving the loss function using Powell, the search range of image parameters such as the yaw angle, elevation, pitch angle, and roll angle of the first orthographic image can be configured according to the operating conditions and parameter accuracy of the corresponding device to narrow the search range as much as possible.

[0118] For example, in a scenario where a drone inspects a photovoltaic power station, the search range of image parameters such as the yaw angle, elevation, pitch angle, and roll angle of the first orthographic image can be specifically configured according to the operating conditions and parameter accuracy of the drone, so as to narrow the search range as much as possible and obtain the yaw angle, elevation, pitch angle, roll angle and other image parameters of the first orthographic image that meet the preset loss conditions more quickly.

[0119] In another embodiment, a genetic algorithm is used to solve the loss function until the function value of the loss function meets a preset loss condition, thereby obtaining the image parameters of the first orthographic image.

[0120] In another embodiment, a particle swarm algorithm is used to solve the loss function until the function value of the loss function meets a preset loss condition, thereby obtaining the image parameters of the first orthographic image.

[0121] After obtaining the image parameters of the first orthographic image, such as the yaw angle, elevation, pitch angle, and roll angle, the obtained image parameters of the first orthographic image, such as the yaw angle, elevation, pitch angle, and roll angle, may be written into the EXIF ​​file of the first orthographic image.

[0122] In an embodiment of the present application, since the first orthophoto image and the second orthophoto image have an image overlapping area, image matching is performed on the first orthophoto image and the second orthophoto image, and the feature point pairs that match between the first orthophoto image and the second orthophoto image can be quickly determined. By using the pixel coordinates of the feature point pairs that match between the first orthophoto image and the second orthophoto image, and the image parameters to be determined of the first orthophoto image and the second orthophoto image, a geographic coordinate function of the mapped geographic points corresponding to the feature point pairs in the geographic coordinate system can be cleverly constructed, so that the geographic coordinate function contains the image parameters to be determined. Since the feature point pairs between the two orthophoto images are determined by image matching, the geographic coordinates of the mapped geographic points corresponding to the feature point pairs are also matched. Based on this, a loss function can be constructed based on the geographic coordinate function of the mapped geographic points corresponding to the feature point pairs. By solving the loss function, the image parameters to be determined that meet the preset loss conditions are obtained. It can be seen that in the image parameter determination method of the present application, the loss function is constructed using the geographic coordinate function of the mapped geographic points of the feature point pair, so that the loss function corresponds to the geographic coordinate function of the mapped geographic points, which contains the image parameters to be determined. Therefore, the loss function is solved to obtain the image parameters to be determined of the first orthographic image that meets the preset loss conditions, which can quickly and accurately determine the image parameters. On this basis, the determined image parameters can be used to stitch and process multiple images of the photovoltaic power station, thereby analyzing the entire area of ​​the photovoltaic power station and locating problems, effectively improving the daily management and maintenance efficiency of the photovoltaic power station.

[0123] like Figure 3 As shown, another embodiment of the present application provides a flowchart of constructing a geographic coordinate function of mapping a first feature point to a geographic point. Figure 3 The difference from other embodiments is that step S203 in other embodiments can be specifically refined as follows: Figure 3 Steps S2031 to S2034 in .

[0124] In step S2031, a first three-dimensional rotation matrix is ​​constructed based on the yaw angle, pitch angle, and roll angle of the first orthographic image. It should be noted that in this step, the yaw angle, pitch angle, and roll angle can all be unknown parameters, or one or more of them can be known parameters.

[0125] The yaw angle, the pitch angle and the roll angle are three Euler angles for describing the rotation state of an object, the yaw angle is used to represent the angle of rotation of the object around the vertical Z axis, the roll angle is used to represent the angle of rotation of the object around the X axis, and the pitch angle is used to represent the angle of rotation of the object around the Y axis. In the scene of the unmanned aerial vehicle inspecting the photovoltaic power station, the yaw angle of the unmanned aerial vehicle can be understood as the included angle between the actual flight direction of the unmanned aerial vehicle and the planned flight direction, the roll angle can be understood as the left-right inclination angle of the unmanned aerial vehicle, and the pitch angle can be understood as the included angle between the front-back direction of the unmanned aerial vehicle and the horizontal plane.

[0126] The three-dimensional rotation matrix can be used to describe the attitude of the image acquisition device relative to the geographic coordinate system, and therefore, the pixel points in the image acquisition device can be accurately mapped into the geographic coordinate system based on the three-dimensional rotation matrix. Based on this, the first three-dimensional rotation matrix can be constructed based on the yaw angle, the pitch angle and the roll angle of the first front-view image.

[0127] In step S2032, a first scaling matrix is constructed based on the altitude of the first front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is an image parameter to be determined. In this step, the parameter value of the altitude can be known or to be determined.

[0128] The altitude is the distance between the image acquisition device and the ground. Due to the different heights between the image acquisition device and the ground, the physical sizes of the pixel points in the photographed front-view image are different.

[0129] Since there can be size differences between different coordinate systems, the size differences between different coordinate systems can be adjusted based on the scaling matrix. Based on this, the first scaling matrix can be constructed based on the altitude of the first front-view image.

[0130] In step S2033, an intrinsic matrix of the image acquisition device is obtained, and the image acquisition device is the image acquisition device for photographing the first front-view image and the second front-view image.

[0131] In the scene of the unmanned aerial vehicle inspecting the photovoltaic power station, the image acquisition device can be an image acquisition device configured on the unmanned aerial vehicle.

[0132] The intrinsic matrix is used to describe the geometric characteristics and optical properties inside the image acquisition device. Therefore, based on the intrinsic matrix, the first feature point can be accurately mapped from the pixel coordinate system to the geographic coordinate system. In an embodiment, the intrinsic matrix can be represented as

[0133]

[0134] wherein, is used to represent the effective focal length of the front-view image on the x axis, for representing the effective focal length of the positive image on the y-axis, and is the principal point coordinate, i.e. for representing the horizontal coordinate of the image center point in the pixel coordinate system, for representing the vertical coordinate of the image center point in the pixel coordinate system.

[0135] In step S2034, a geographical coordinate function of the mapping geographical point of the first feature point is constructed based on the pixel coordinate of the first feature point in the feature point pair, the intrinsic matrix, the first three-dimensional rotation matrix and the first scaling matrix.

[0136] In an embodiment, the geographical coordinate function of the mapping geographical point of the first feature point can also be constructed based on the pixel coordinate of the first feature point in the feature point pair, the inverse matrix of the intrinsic matrix, the first three-dimensional rotation matrix and the first scaling matrix.

[0137] For example, if the image parameters of the first positive image are to be determined, and the first three-dimensional rotation matrix is constructed based on the yaw angle, the pitch angle and the roll angle of the first positive image represents the first positive image, the second positive image is represented by , the first positive image , the first feature point in the first positive image , and the second feature point in the second positive image is a feature point pair. The process of constructing the geographical coordinate function of the mapping geographical point of the first feature point based on the pixel coordinate of the first feature point, the inverse matrix of the intrinsic matrix, the first three-dimensional rotation matrix and the first scaling matrix is as follows.

[0138] The first three-dimensional rotation matrix is constructed based on the yaw angle, the pitch angle and the roll angle of the first positive image , and the first three-dimensional rotation matrix can be represented as

[0139] =

[0140] wherein, for representing the first three-dimensional rotation matrix, for representing the yaw angle of the first positive image , for representing the pitch angle of the first positive image , for representing the roll angle of the first positive image , for representing the roll angle of the first positive image , for representing the roll angle of the first positive image the pitch angle. It should be noted that the image parameters of known parameter values can be substituted with specific values, for example, if only the parameter value of the yaw angle needs to be determined, the angle values of the pitch angle and the roll angle (the ideal angle value is zero under the premise of a front-facing image) are substituted into the first three-dimensional rotation matrix; if only the parameter value of the pitch angle needs to be determined, the angle values of the yaw angle and the roll angle are substituted into the first three-dimensional rotation matrix; if only the parameter value of the roll angle needs to be determined, the angle values of the yaw angle and the pitch angle are substituted into the first three-dimensional rotation matrix; if the parameter values of two angles need to be determined, the remaining angle values are substituted into the first three-dimensional rotation matrix; if the parameter values of three angles need to be determined, the representation form of the first three-dimensional rotation matrix is used.

[0141] based on the elevation of the first front-facing image , a first scaling matrix is constructed, which can be represented as

[0142] =

[0143] wherein, is used to represent the first scaling matrix, is used to represent the elevation of the first front-facing image , and if the parameter value of the elevation is known, the specific parameter value can be substituted into the first scaling matrix.

[0144] based on the first feature point of the first front-facing image , the inverse matrix of the intrinsic matrix , the first three-dimensional rotation matrix , and the first scaling matrix , a geographical coordinate function of a mapped geographical point of the first feature point is constructed, and the geographical coordinate function of the mapped geographical point of the first feature point can be represented as

[0145] =

[0146] wherein, is used to represent the inverse matrix of the intrinsic matrix, is used to represent the geographical coordinate function of the mapped geographical point of the first feature point .

[0147] based on the pixel coordinates of the first feature point , the inverse matrix of the intrinsic matrix , the first three-dimensional rotation matrix , and the first scaling matrix , a geographical coordinate function of a mapped geographical point of the first feature point is constructed In the process, due to the inverse matrix of the internal parameter matrix , the first three-dimensional rotation matrix and the first scaling matrix They are all three-dimensional matrices, so the first feature point obtained is The geographic coordinate function of the mapped geographic point The geographic coordinates represented are also three-dimensional. The two-dimensional pixel coordinates are converted to the three-dimensional geographic coordinates of the mapped geographic point. During the conversion process, the elevation axis of the three-dimensional geographic coordinates of the mapped geographic point is inaccurate. Therefore, in order to reduce the impact on the accuracy of the image parameters, in one embodiment, the first feature point can also be The geographic coordinate function of the mapped geographic point Normalize the elevation axis of the first feature point The three-dimensional geographic coordinates of the mapped geographic point are converted into two-dimensional geographic coordinates, that is,

[0148] =

[0149] in, Used to represent the first feature point The two-dimensional geographic coordinates of the mapped geographic points, Used to represent the first feature point The horizontal coordinate of the three-dimensional geographic coordinates of the mapped geographic point, Used to represent the first feature point The vertical coordinate of the three-dimensional geographic coordinates of the mapped geographic point, Used to represent the first feature point The elevation coordinates of the mapped geographic point in three-dimensional geographic coordinates.

[0150] By using the intrinsic parameter matrix of the image acquisition device, the first three-dimensional rotation matrix and the first scaling matrix, the first feature point is mapped from the pixel coordinate system to the geographic coordinate system, so as to accurately obtain the mapped geographic point corresponding to the first feature point and more accurately construct the geographic coordinate function of the mapped geographic point of the first feature point.

[0151] like Figure 4 As shown, another embodiment of the present application provides a flowchart of constructing a geographic coordinate function of a mapping geographic point of a second feature point. Figure 4 The difference from other embodiments is that step S203 in other embodiments can be specifically refined as follows: Figure 4 Steps S2035 to S2038 in .

[0152] In step S2035, a second three-dimensional rotation matrix is constructed based on the yaw angle, the pitch angle and the roll angle of the second front-view image. It should be noted that in this step, the yaw angle, the pitch angle and the roll angle can all be unknown parameters, or one or more of them can be known parameters.

[0153] The three-dimensional rotation matrix can be used to describe the pose of the image acquisition device relative to the geographic coordinate system, and therefore, the pixel points in the image acquisition device can be accurately mapped into the geographic coordinate system based on the three-dimensional rotation matrix. Based on this, the second three-dimensional rotation matrix can be constructed based on the yaw angle, the pitch angle and the roll angle of the second front-view image.

[0154] In step S2036, a second scaling matrix is constructed based on the altitude of the second front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is an image parameter to be determined. In this step, the parameter value of the altitude can be known or to be determined.

[0155] Since there can be size differences between different coordinate systems, the size differences between different coordinate systems can be adjusted based on the scaling matrix. Based on this, the second scaling matrix can be constructed based on the altitude of the second front-view image.

[0156] In step S2037, an intrinsic matrix of the image acquisition device is obtained, and the image acquisition device is the image acquisition device that captures the first front-view image and the second front-view image.

[0157] In the scenario of unmanned aerial vehicle inspection of photovoltaic power stations, the image acquisition device can be an image acquisition device configured on the unmanned aerial vehicle.

[0158] In step S2038, a geographic coordinate function of a mapped geographic point of the second feature point is constructed based on the intrinsic matrix, the pixel coordinates of the second feature point in the feature point pair, the second three-dimensional rotation matrix and the second scaling matrix.

[0159] The translation matrix can represent the positional offset of the image acquisition device relative to the geographic coordinate system. In the embodiment in which the geographic coordinates of the first front-view image and the second front-view image are geographic coordinates in the coordinate system of the image acquisition device itself rather than in the geodetic coordinate system, since the shooting positions of the first front-view image and the second front-view image are different, the geographic coordinates of the common features between the first front-view image and the second front-view image are deviated. Therefore, in order to more accurately determine the geographic coordinate function of the mapped geographic point of the second feature point, in one embodiment, a translation matrix can be constructed based on the geographic coordinate deviation between the first front-view image and the second front-view image, and the geographic coordinate function of the mapped geographic point of the second feature point is constructed by mapping the second feature point from the image coordinate system to the geographic coordinate system using the translation matrix, the inverse matrix of the intrinsic matrix, the second three-dimensional rotation matrix and the second scaling matrix.

[0160] For example, if the image parameters of the first equirectangular image are to be determined, and the first equirectangular image is represented as the second equirectangular image is represented as the first equirectangular image the first feature point in the first equirectangular image the second feature point in the second equirectangular image

[0161] the geographical coordinate deviation between the first equirectangular image and the second equirectangular image is determined, and the geographical coordinate deviation can be represented as

[0162]

[0163] wherein, represents the horizontal coordinate of the first equirectangular image in the geographical coordinate system, represents the vertical coordinate of the first equirectangular image in the geographical coordinate system, represents the horizontal coordinate of the second equirectangular image in the geographical coordinate system, represents the vertical coordinate of the second equirectangular image in the geographical coordinate system, represents the horizontal coordinate deviation between the first equirectangular image and the second equirectangular image , represents the vertical coordinate deviation between the first equirectangular image and the second equirectangular image .

[0164] based on the geographical coordinate deviation between the first equirectangular image and the second equirectangular image , a translation matrix

[0165] =

[0166] wherein, represents the translation matrix.

[0167] based on the yaw angle, the pitch angle and the roll angle of the second equirectangular image , a second three-dimensional rotation matrix is constructed, and the second three-dimensional rotation matrix can be represented as

[0168]

[0169]

[0170] in, Used to represent the second three-dimensional rotation matrix, Used to represent the second orthographic image The yaw angle, Used to represent the second orthographic image The roll angle, Used to represent the second orthographic image The pitch angle. It should be noted that the image parameters with known parameter values ​​can be substituted into specific numerical values. For example, if only the parameter value of the yaw angle needs to be determined, the angle values ​​of the pitch angle and the roll angle (under the premise of the orthographic image, the ideal angle value is zero) are substituted into the second three-dimensional rotation matrix; if only the parameter value of the pitch angle needs to be determined, the angle values ​​of the yaw angle and the roll angle are substituted into the second three-dimensional rotation matrix; if only the parameter value of the roll angle needs to be determined, the angle values ​​of the yaw angle and the pitch angle are substituted into the second three-dimensional rotation matrix; if the parameter values ​​of two angles need to be determined, the remaining angle values ​​are substituted into the second three-dimensional rotation matrix; if the parameter values ​​of three angles need to be determined, the above-mentioned representation of the second three-dimensional rotation matrix is ​​used.

[0171] Based on the second orthographic image The second scaling matrix is ​​constructed based on the elevation of

[0172] =

[0173] in, Used to represent the second scaling matrix, Used to represent the second orthographic image If the parameter value of the elevation is known, the specific parameter value can be substituted into the first scaling matrix.

[0174] Based on the translation matrix , the inverse matrix of the internal parameter matrix , the second three-dimensional rotation matrix and the second scaling matrix , the second feature point Map to geographic coordinates and construct the second feature point The geographic coordinate function of the mapped geographic point , the second feature point The geographic coordinate function of the mapped geographic point can be expressed as

[0175] =

[0176] in, The inverse matrix of the intrinsic parameter matrix.

[0177] Based on the translation matrix , the inverse matrix of the internal parameter matrix , the second three-dimensional rotation matrix and the second scaling matrix , the second feature point Map to geographic coordinates and construct the second feature point The geographic coordinate function of the mapped geographic point In the process, due to the inverse matrix of the internal parameter matrix , the second three-dimensional rotation matrix , the second scaling matrix and the translation matrix They are all three-dimensional matrices, so the second characteristic point obtained is The geographic coordinate function of the mapped geographic point The geographic coordinates represented are also three-dimensional. The two-dimensional pixel coordinates are converted to the three-dimensional geographic coordinates of the mapped geographic point. During the conversion process, the elevation axis of the three-dimensional geographic coordinates of the mapped geographic point is inaccurate. Therefore, in order to reduce the impact on the accuracy of the image parameters, in one embodiment, the second feature point can also be The geographic coordinate function of the mapped geographic point Normalize the elevation axis of the second feature point The three-dimensional geographic coordinates of the mapped geographic point are converted into two-dimensional geographic coordinates, that is,

[0178] =

[0179] in, Used to represent the second feature point The two-dimensional geographic coordinates of the mapped geographic points, Used to represent the second feature point The horizontal coordinate of the three-dimensional geographic coordinates of the mapped geographic point, Used to represent the second feature point The vertical coordinate of the three-dimensional geographic coordinates of the mapped geographic point, Used to represent the second feature point The elevation coordinates of the mapped geographic point in three-dimensional geographic coordinates.

[0180] By using the intrinsic parameter matrix of the image acquisition device, the second three-dimensional rotation matrix and the second scaling matrix, the second feature point is mapped to the geographic coordinates, so that the mapped geographic point of the second feature point can be accurately obtained, and the geographic coordinate function of the mapped geographic point of the second feature point can be constructed more cleverly.

[0181] It should be noted that the translation matrix can also be used to construct the geographic coordinate function of the mapped geographic point of the first feature point. The specific construction method can refer to the geographic coordinate function of the mapped geographic point of the second feature point. In some embodiments, the geographic coordinate function of the mapped geographic points of the first and second feature points can be constructed without considering the translation matrix. Instead, when calculating the loss function, that is, the geographic distance between the mapped geographic points, the geographic coordinates of one feature point are added to or subtracted from the geographic coordinate deviation, and then the geographic distance is calculated with the geographic coordinates of the other feature point.

[0182] It can be understood that the embodiment of constructing the geographic coordinate function in the present application is not limited to the form of the above three-dimensional rotation matrix and scaling matrix. If other types of image parameters need to be determined, a transformation matrix for transforming pixel coordinates into geographic coordinates can be constructed based on the mapping relationship between the image parameters and the geographic coordinates.

[0183] like Figure 5 As shown, another embodiment of the present application provides a flowchart for constructing a loss function based on a geographic coordinate function of a mapped geographic point of a feature point pair. Figure 5 The difference from other embodiments is that step S204 in other embodiments can be specifically refined as follows: Figure 5 Steps S2041 and S2042 in .

[0184] In step S2041, based on the geographic coordinate function of the mapped geographic points of the feature point pair between the first orthographic image and any second orthographic image, a sub-loss function corresponding to the feature point pair is constructed, and the sub-loss function is related to the distance between the mapped geographic points of the feature point pair.

[0185] The sub-loss function can be used to represent the distance between the mapped geographic points of a feature point pair. The sub-loss function can be expressed as

[0186] =

[0187] in, Used to represent the sub-loss function, Used to represent the transpose operator.

[0188] In step S2042 , a loss function is constructed based on sub-loss functions of feature point pairs between the first orthographic image and the plurality of second orthographic images.

[0189] In the case that there are multiple second positive images, a loss function is constructed based on the sub-loss function of the feature point pairs between the first positive image and the multiple second positive images, so that the constructed loss function can be used to reflect the geographical distance between the feature points in the first positive image and the corresponding feature points in each second positive image, so that the constructed loss function is more accurate, and further makes the image parameters of the subsequent obtained first positive image more accurate.

[0190] In some examples, constructing the loss function based on the sub-loss function of the feature point pairs between the first positive image and the multiple second positive images includes: determining the total number of the feature point pairs between the first positive image and the multiple second positive images; and constructing the loss function based on the sub-loss function of the feature point pairs between the first positive image and the multiple second positive images and the total number of the feature point pairs.

[0191] Based on the sub-loss function of the feature point pairs between the first positive image and the multiple second positive images and the total number of the feature point pairs, a loss function in an average state can be constructed, so that in the subsequent solving process of the loss function, the loss function can quickly reach a convergence state, and thus the image parameters corresponding to the optimal state of the first positive image can be quickly solved.

[0192] In an embodiment, the loss function between the first positive image and the second positive image is constructed based on the sub-loss function of all the feature point pairs between the first positive image and the second positive image, and the loss function is

[0193]

[0194] wherein, is used to represent the loss function between the first positive image and the second positive image.

[0195] The average loss function between the first positive image and the second positive image is constructed based on the total number of the feature point pairs between the first positive image and the second positive image, and the average loss function is

[0196]

[0197] wherein, is used to represent the average loss function between the first positive image and the second positive image.

[0198] The loss function is constructed based on the average loss function between the first positive image and each second positive image.

[0199] For example, in the case that there is one first positive image and three second positive images, the first positive image is represented as ​​, the three second orthographic images are represented as 、 as well as , and There is feature point pairs, and There is feature point pairs, and There is feature point pairs.

[0200] based on and between The sub-loss function of feature point pairs is constructed and The loss function between ,based on and between The sub-loss function of feature point pairs is constructed and The loss function between ,based on and between The sub-loss function of feature point pairs is constructed and The loss function between .

[0201] based on and Loss function between And the number of feature point pairs ,get and The average loss function between ,based on and Loss function between And the number of feature point pairs ,get and The average loss function between ,based on and Loss function between And the number of feature point pairs ,get and The average loss function between .

[0202] based on and the average loss function between , and the average loss function between and the average loss function between and , the loss function E can be obtained, which can be represented as

[0203] + +

[0204] In another embodiment, the loss function between the first positive image and a second positive image is constructed based on the sub-loss function of all feature point pairs between the first positive image and the second positive image; the loss function between the first positive image and multiple second positive images is constructed based on the loss function between the first positive image and each second positive image; and the loss function is obtained by averaging the loss functions between the first positive image and multiple second positive images based on the total number of feature point pairs between the first positive image and multiple second positive images.

[0205] For example, in the case of one first positive image and three second positive images, the first positive image is represented as , the three second positive images are represented as , and , and have feature point pairs, and have feature point pairs, and have feature point pairs.

[0206] The loss function between and is constructed based on the sub-loss function of feature point pairs between and , the loss function between and is constructed based on the sub-loss function of feature point pairs between and , and the loss function between and is constructed based on the sub-loss function of feature point pairs between and . loss function between loss function between .

[0207] based on loss function between loss function between , loss function between loss function between and loss function between loss function between , obtain loss function between , and loss function between + + .

[0208] based on loss function between feature point pairs between loss function between feature point pairs between loss function between feature point pairs between obtain total number of feature point pairs between and = + + .

[0209] loss function between , and loss function between and obtain loss function E, and the loss function E can be expressed as

[0210] E=

[0211] In some examples, the method for determining image parameters provided by the present application further comprises: before image matching of the first orthographic image and the second orthographic image, and before determining the matched feature point pairs between the first orthographic image and the second orthographic image, performing distortion correction on the first orthographic image and the second orthographic image to obtain the first orthographic image and the second orthographic image after distortion correction, and the first orthographic image and the second orthographic image after distortion correction are used for extracting feature points.

[0212] ​​​​By performing distortion correction on the first and second orthographic images, geometric distortion introduced during the imaging process can be eliminated, further ensuring that feature points of the distortion-corrected first orthographic image and multiple second orthographic images can be extracted more accurately subsequently.

[0213] In one embodiment, the distortion correction can be performed on the first orthographic image and the second orthographic image by the following steps, which specifically include a first determination step, a distortion correction step, and a distortion correction processing step, wherein:

[0214] The first determination step: determine the internal parameter matrix K, which can be expressed as

[0215]

[0216] in, Used to indicate the effective focal length of the orthographic image on the x-axis. Used to represent the effective focal length of the orthographic image on the y-axis, and is the principal point coordinate, that is Used to represent the horizontal coordinate of the center point of the image in the pixel coordinate system. Used to represent the vertical coordinate of the center point of the image in the pixel coordinate system.

[0217] Distortion correction steps: Image distortion correction coefficients usually include radial distortion coefficients and tangential distortion coefficients. Among them, the radial distortion coefficient includes , as well as , the tangential distortion coefficients include and Based on this, the distortion model of the orthographic image can be expressed as

[0218] (1+ + )+2 xy+ )

[0219] (1+ + )+ xy+ )

[0220] in,( ) is used to represent the pixel coordinates of the pixel point in the orthographic image after distortion correction in the pixel coordinate system. It is used to represent the horizontal coordinate of the pixel point in the orthographic image after distortion correction in the pixel coordinate system. y(x, y) is used for representing the vertical coordinate of a pixel point in the front-view image after distortion correction in a pixel coordinate system, (x, y) is the pixel coordinate of a pixel point in the front-view image before distortion correction in the pixel coordinate system, x is used for representing the horizontal coordinate of a pixel point in the front-view image before distortion correction in the pixel coordinate system, and y is used for representing the vertical coordinate of a pixel point in the front-view image before distortion correction in the pixel coordinate system, .

[0221] The distortion correction processing step is: constructing a mapping matrix, and iteratively interpolating the pixel coordinates of the pixel points in the front-view image before distortion correction in the distortion correction step and the pixel coordinates of the corresponding pixel points after distortion correction, so as to obtain the mapping matrix of the front-view image before distortion correction and after distortion correction, that is, to obtain the mapping table of the distortion coordinates and the non-distortion coordinates of the front-view image. Then, based on the mapping table, the pixel points in the front-view image are one-to-one mapped to obtain the corresponding image after distortion correction.

[0222] In order to facilitate understanding, the processing process of the image parameter determination method of the present application is described below by taking the unmanned aerial vehicle performing inspection on the photovoltaic power station as an example. Figure 6 The flowchart of the image parameter determination method of the present application in the scenario of the unmanned aerial vehicle performing inspection on the photovoltaic power station is provided. The image parameter determination method of the present application in the scenario of the unmanned aerial vehicle performing inspection on the photovoltaic power station comprises steps S601 to S610.

[0223] In step S601, the unmanned aerial vehicle performs inspection on the photovoltaic power station and photographs n front-view images, that is, a front-view image set .

[0224] In step S602, a first front-view image and a plurality of second front-view images are determined from the plurality of front-view images.

[0225] When the image parameters of are determined, the first front-view image can be , and the second front-view image having an image overlapping area with can be determined from . In order to facilitate understanding, it is assumed that the geographic distance between and each front-view image in satisfies a preset condition, and then each front-view image in is determined as a second front-view image, that is, is the first front-view image, and each front-view image in is a second front-view image.

[0226] In step S603, the first front-view image and the plurality of second front-view images are subjected to distortion correction.

[0227] ​In step S604, image matching is performed on the first front-view image and each second front-view image to determine a feature point pair between the first front-view image and each second front-view image.

[0228] Steps S605 and S606 are described by taking one feature point pair as an example to solve a function of a geographic coordinate of a mapped geographic point of a first feature point in the feature point pair and a function of a geographic coordinate of a mapped geographic point of a second feature point in the feature point pair.

[0229] In step S605, a function of a geographic coordinate of a mapped geographic point of the first feature point in the feature point pair is constructed based on a pixel coordinate of the first feature point in the feature point pair and to-be-determined image parameters of the first front-view image.

[0230] A first three-dimensional rotation matrix is constructed based on a yaw angle, a pitch angle and a roll angle of the first front-view image.

[0231] A first scaling matrix is constructed based on an altitude of the first front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is a to-be-determined image parameter.

[0232] A function of a geographic coordinate of a mapped geographic point of the first feature point is constructed based on the pixel coordinate of the first feature point of the first front-view image, an inverse matrix of an intrinsic matrix, the first three-dimensional rotation matrix and the first scaling matrix, and a three-dimensional geographic coordinate represented by the function of the geographic coordinate of the mapped geographic point of the first feature point is converted into a two-dimensional geographic coordinate by normalizing an altitude axis in the function of the geographic coordinate of the mapped geographic point of the first feature point.

[0233] In step S606, a function of a geographic coordinate of a mapped geographic point of the second feature point in the feature point pair is constructed based on a pixel coordinate of the second feature point in the feature point pair, to-be-determined image parameters of the second front-view image and a geographic coordinate deviation.

[0234] The geographic coordinate deviation is determined based on a first geographic coordinate of the first front-view image and a second geographic coordinate of the second front-view image.

[0235] A translation matrix is constructed based on the geographic coordinate deviation between the first front-view image and the second front-view image.

[0236] A second three-dimensional rotation matrix is constructed based on a yaw angle, a pitch angle and a roll angle of the second front-view image.

[0237] A second scaling matrix is constructed based on an altitude of the second front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is a to-be-determined image parameter.

[0238] The mapping geographical point geographical coordinate function of the second feature point is constructed based on the translation matrix, the inverse matrix of the intrinsic matrix, the second three-dimensional rotation matrix and the second scaling matrix, and the three-dimensional geographical coordinates represented by the mapping geographical point geographical coordinate function of the second feature point are converted into two-dimensional geographical coordinates by normalizing the elevation axis in the mapping geographical point geographical coordinate function of the second feature point.

[0239] In step S607, a sub-loss function is constructed based on the mapping geographical point geographical coordinate function of the feature point pair.

[0240] In step S608, a loss function is constructed based on the sub-loss function of the feature point pair and the total number of feature point pairs between the first front image and the plurality of second front images.

[0241] In step S609, a nonlinear programming algorithm is used to solve the image parameters to be determined of the first front image under the condition that the loss function meets the preset loss condition.

[0242] In the solving process, first, the initial value of the image parameters of each front image is obtained, and the optimal solution is searched from the initial value until the loss function reaches a convergence state to obtain the image parameters of the first front image. As an example, the initial value of the image parameters can be specified or obtained from the image acquisition device, for example, from the EXIF file of the image.

[0243] In step S610, the determined image parameters of the first front image are written into the EXIF file of the first front image.

[0244] When the image parameters of other front images in the front image set I need to be determined, steps S602 to S610 can be repeatedly executed. } can be repeatedly executed.

[0245] It should be noted that the image parameter determination device corresponds to the above-mentioned image parameter determination method, and all implementation manners in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects, which will not be repeated here.

[0246] Based on the same inventive concept, the embodiments of the present application also provide an image parameter determination device. Specifically, the image parameter determination device is described in combination with Figure 7 The image parameter determination device provided by the embodiments of the present application is described in detail.

[0247] Figure 7 is a structural schematic diagram of an image parameter determination device provided by the embodiments of the present application.

[0248] As Figure 7As shown, the image parameter determination apparatus 700 can include an acquisition unit 710, an image matching unit 720, a first construction unit 730, a second construction unit 740, and a calculation unit 750.

[0249] The acquisition unit 710 can be configured to acquire a first orthographic image and a second orthographic image, the first orthographic image and the second orthographic image having an image overlapping area.

[0250] The image matching unit 720 can be configured to perform image matching on the first orthographic image and the second orthographic image, and determine a matching feature point pair between the first orthographic image and the second orthographic image.

[0251] The first construction unit 730 can be configured to construct a geographical coordinate function of a mapping geographical point of the feature point pair based on pixel coordinates of the feature point pair, and the image parameters to be determined of the first orthographic image and the second orthographic image.

[0252] The second construction unit 740 can be configured to construct a loss function based on the geographical coordinate function of the mapping geographical point of the feature point pair.

[0253] The calculation unit 750 can be configured to solve the image parameters to be determined of the first orthographic image under a condition that the loss function meets a preset loss condition.

[0254] In an embodiment, the second construction unit can be further configured to determine a geographical coordinate deviation between the first orthographic image and the second orthographic image based on a first geographical coordinate of the first orthographic image and a second geographical coordinate of the second orthographic image, the loss function being related to a distance between the loss function and the mapping geographical point of the feature point pair; and construct the loss function based on the geographical coordinate deviation and the geographical coordinate function of the mapping geographical point of the feature point pair.

[0255] In an embodiment, the second construction unit can be further configured to determine the first geographical coordinate based on a latitude and longitude parameter of the first orthographic image; determine the second geographical coordinate based on a latitude and longitude parameter of the second orthographic image; and determine the geographical coordinate deviation between the first orthographic image and the second orthographic image based on the first geographical coordinate and the second geographical coordinate.

[0256] In an embodiment, the image parameters to be determined include at least one of a yaw angle, a pitch angle, a roll angle, and an altitude.

[0257] In an embodiment, the first constructing unit is further configured to construct a first three-dimensional rotation matrix based on a yaw angle, a pitch angle and a roll angle of the first front-view image, and construct a first scaling matrix based on an altitude of the first front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is an image parameter to be determined; obtain an intrinsic matrix of an image acquisition device, the image acquisition device being an image acquisition device used to capture the first front-view image and the second front-view image; and construct a geographical coordinate function of a mapped geographical point of the first feature point based on the pixel coordinate of the first feature point in the feature point pair, the intrinsic matrix, the first three-dimensional rotation matrix and the first scaling matrix.

[0258] In an embodiment, the first constructing unit is further configured to construct a second three-dimensional rotation matrix based on a yaw angle, a pitch angle and a roll angle of the second front-view image, and construct a second scaling matrix based on an altitude of the second front-view image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the altitude is an image parameter to be determined; obtain an intrinsic matrix of an image acquisition device, the image acquisition device being an image acquisition device used to capture the first front-view image and the second front-view image; and construct a geographical coordinate function of a mapped geographical point of the second feature point based on the pixel coordinate of the second feature point in the feature point pair, the intrinsic matrix, the second three-dimensional rotation matrix and the second scaling matrix.

[0259] In an embodiment, the second constructing unit is further configured to construct a sub-loss function corresponding to the feature point pair based on the geographical coordinate function of the mapped geographical point of the feature point pair between the first front-view image and any second front-view image, the sub-loss function being related to a distance between the mapped geographical point of the feature point pair; and construct the loss function based on the sub-loss functions of the feature point pairs between the first front-view image and the plurality of second front-view images.

[0260] In an embodiment, the second constructing unit is further configured to determine a total number of the feature point pairs between the first front-view image and the plurality of second front-view images; and construct the loss function based on the total number of the feature point pairs and the sub-loss functions of the feature point pairs between the first front-view image and the plurality of second front-view images.

[0261] In an embodiment, the obtaining unit is further configured to determine, from the plurality of front-view images, a second front-view image having an image overlapping area with the first front-view image.

[0262] In an embodiment, the obtaining unit is further configured to determine a geographical distance between the first front-view image and each front-view image based on the geographical coordinates of the first front-view image and the front-view images; determine whether the geographical distance meets a preset condition; and determine the front-view image as the second front-view image in a case where the geographical distance meets the preset condition.

[0263] In one embodiment, the image parameter determination apparatus of the present application further comprises a distortion correction unit. The distortion correction unit can be configured to perform distortion correction on the first front-view image and the second front-view image before image matching is performed on the first front-view image and the second front-view image, and before the matching feature point pairs between the first front-view image and the second front-view image are determined, to obtain a distortion corrected first front-view image and a distortion corrected second front-view image, and the distortion corrected first front-view image and the distortion corrected second front-view image are used for feature point extraction.

[0264] Figure 8 A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0265] The electronic device can include a processor 801 and a memory 802 having computer program instructions stored therein.

[0266] Specifically, the processor 801 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0267] The memory 802 can include a mass storage for data or instructions. By way of example and not limitation, the memory 802 can include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive or a combination of two or more of these. The memory 802 can include removable or non-removable (or fixed) media, where appropriate. The memory 802 can be internal or external to the integrated gateway disaster recovery device, as appropriate. In certain embodiments, the memory 802 is non-volatile, solid-state memory.

[0268] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform operations described with reference to the methods according to the aspects of the present application.

[0269] The processor 801 reads and executes the computer program instructions stored in the memory 802 to implement any one of the image parameter determination methods in the above embodiments.

[0270] In one example, the electronic device can further include a communication interface 803 and a bus 810. As shown in Figure 8 the processor 801, the memory 802, and the communication interface 803 are connected through the bus 810 and complete communication therebetween.

[0271] The communication interface 803 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0272] The bus 810 includes hardware, software or both to couple components of the online data traffic billing device to each other. By way of example, and not limitation, the bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where suitable, the bus 810 can include one or more buses. Although particular buses are described and shown in the embodiments of the present application, the present application contemplates any suitable bus or interconnect.

[0273] In addition, in combination with the image parameter determination method in the above embodiments, the embodiments of the present application can provide a computer storage medium for implementation. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to implement any of the image parameter determination methods in the above embodiments.

[0274] The embodiments of the present application also provide a computer program product, instructions in the computer program product are executed by a processor of an electronic device to make the electronic device execute the image parameter determination method provided by the embodiments of the present application.

[0275] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of well-known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0276] The functions shown in the above-described structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. The "machine-readable medium" can include any medium that can store or transfer information. Examples of the machine-readable medium include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber medium, radio frequency (RF) links, and the like. The code segments can be downloaded via a computer network such as the Internet, an intranet, and the like. It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in an order different from the embodiments, or several steps can be performed simultaneously.

[0277] The computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These instructions can be stored in a computer- readable medium, which can be a storage device or memory element associated with the computer or other programmable data processing apparatus. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operations to be performed on the computer or other programmable data processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable data processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0278] The above-described are merely specific implementations of the present application, and those skilled in the art can clearly understand the specific working processes of the above-described systems, modules and units in the description for the sake of convenience and brevity. The protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for determining image parameters, characterized in that: include: Acquire a first orthographic image and a second orthographic image captured by the drone using an image acquisition device during an inspection process, wherein the first orthographic image and the second orthographic image have an image overlap area, and the first orthographic image and the second orthographic image are shot at different locations; performing image matching on the first orthographic image and the second orthographic image to determine matching feature point pairs between the first orthographic image and the second orthographic image; constructing a geographic coordinate function of the mapped geographic points of the feature point pair in the image acquisition device coordinate system based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first orthographic image and the second orthographic image; Constructing a loss function based on the geographic coordinate function of the mapped geographic points of the feature point pair, wherein the loss function is related to the distance between the mapped geographic points of the feature point pair, wherein the loss function is constructed based on the geographic coordinate function of the mapped geographic points of the feature point pair, comprising: determining a geographic coordinate deviation between the first orthographic image and the second orthographic image based on first geographic coordinates of the first orthographic image and second geographic coordinates of the second orthographic image, and constructing the loss function based on the geographic coordinate deviation and the geographic coordinate function of the mapped geographic points of the feature point pair, wherein the first geographic coordinates of the first orthographic image and the second geographic coordinates of the second orthographic image are determined based on shooting conditions recorded by an image acquisition device; Solving the to-be-determined image parameters of the first orthographic image when the loss function satisfies a preset loss condition; The step of constructing a geographic coordinate function of a mapped geographic point of the feature point pair based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first orthographic image and the second orthographic image comprises: constructing a first three-dimensional rotation matrix based on the yaw angle, the pitch angle, and the roll angle of the first orthographic image; constructing a first scaling matrix based on the elevation of the first orthographic image, wherein at least one of the yaw angle, the pitch angle, the roll angle, and the elevation is the image parameter to be determined; Obtaining an intrinsic parameter matrix of an image acquisition device, where the image acquisition device is an image acquisition device that captures the first orthographic image and the second orthographic image; constructing a geographic coordinate function of a mapping geographic point to the first feature point based on the pixel coordinates of the first feature point in the feature point pair, the intrinsic parameter matrix, the first three-dimensional rotation matrix, and the first scaling matrix; The method further comprises: constructing a geographic coordinate function of a mapped geographic point of the feature point pair based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first orthographic image and the second orthographic image; constructing a second three-dimensional rotation matrix based on the yaw angle, pitch angle, and roll angle of the second orthographic image; constructing a second scaling matrix based on the elevation of the second orthographic image, wherein at least one of the yaw angle, the pitch angle, the roll angle, and the elevation is the image parameter to be determined; Obtaining an intrinsic parameter matrix of an image acquisition device, where the image acquisition device is an image acquisition device that captures the first orthographic image and the second orthographic image; A geographic coordinate function of mapping the second feature point to a geographic point is constructed based on the intrinsic parameter matrix, the pixel coordinates of the second feature point in the feature point pair, the second three-dimensional rotation matrix, and the second scaling matrix.

2. The method according to claim 1, characterized in that Determining a geographic coordinate deviation between the first orthographic image and the second orthographic image based on first geographic coordinates of the first orthographic image and second geographic coordinates of the second orthographic image includes: determining the first geographic coordinates based on the latitude and longitude parameters of the first orthographic image; determining the second geographic coordinates based on the latitude and longitude parameters of the second orthographic image; A geographic coordinate deviation between the first orthographic image and the second orthographic image is determined based on the first geographic coordinate and the second geographic coordinate.

3. The method according to claim 1 or 2, characterized in that Constructing a loss function based on the geographic coordinate function of the mapped geographic point of the feature point pair, comprising: constructing, based on a geographic coordinate function of the mapped geographic points of the feature point pair between the first orthographic image and any one of the second orthographic images, a sub-loss function corresponding to the feature point pair, wherein the sub-loss function is related to the distance between the mapped geographic points of the feature point pair; The loss function is constructed based on the sub-loss functions of the feature point pairs between the first orthographic image and a plurality of the second orthographic images.

4. The method according to claim 3, characterized in that Constructing the loss function based on the sub-loss function of the feature point pairs between the first orthographic image and the plurality of second orthographic images includes: determining a total number of the feature point pairs between the first orthographic image and a plurality of the second orthographic images; The loss function is constructed based on the sub-loss functions of the feature point pairs between the first orthographic image and the plurality of second orthographic images and the total number of the feature point pairs.

5. The method according to claim 1 or 2, characterized in that The acquiring of the first orthographic image and the second orthographic image comprises: A second orthographic image having an image overlap area with the first orthographic image is determined from the plurality of orthographic images.

6. The method according to claim 5, characterized in that The determining, from the plurality of orthographic images, the second orthographic image having an image overlapping area with the first orthographic image comprises: determining a geographical distance between the first orthographic image and each of the orthographic images based on the geographical coordinates of the first orthographic image and each of the orthographic images; Determining whether the geographical distance meets a preset condition; When the geographical distance satisfies the preset condition, the orthographic image is determined as the second orthographic image.

7. The method according to claim 1 or 2, characterized in that Before performing image matching on the first orthographic image and the second orthographic image to determine matching feature point pairs between the first orthographic image and the second orthographic image, the method further includes: Distortion correction is performed on the first orthographic image and the second orthographic image to obtain the first orthographic image and the second orthographic image after distortion correction. The first orthographic image and the second orthographic image after distortion correction are used to extract the feature points.

8. A device for determining image parameters, characterized in that: include: an acquisition unit, configured to acquire a first orthographic image and a second orthographic image acquired by the drone using an image acquisition device during an inspection process, wherein the first orthographic image and the second orthographic image have an image overlap area and are shot at different locations; an image matching unit, configured to perform image matching on the first orthographic image and the second orthographic image, and determine matching feature point pairs between the first orthographic image and the second orthographic image; a first constructing unit, configured to construct a geographic coordinate function of the mapped geographic points of the feature point pair in the image acquisition device coordinate system based on the pixel coordinates of the feature point pair and the image parameters to be determined of the first orthographic image and the second orthographic image; a second construction unit, configured to construct a loss function based on the geographic coordinate function of the mapped geographic points of the feature point pair, wherein the loss function is related to the distance between the mapped geographic points of the feature point pair, and the construction of the loss function based on the geographic coordinate function of the mapped geographic points of the feature point pair comprises: determining a geographic coordinate deviation between the first orthographic image and the second orthographic image based on a first geographic coordinate of the first orthographic image and a second geographic coordinate of the second orthographic image, and constructing the loss function based on the geographic coordinate deviation and the geographic coordinate function of the mapped geographic points of the feature point pair, wherein the first geographic coordinate of the first orthographic image and the second geographic coordinate of the second orthographic image are determined based on shooting conditions recorded by an image acquisition device; a calculation unit, configured to solve the to-be-determined image parameter of the first orthographic image when the loss function satisfies a preset loss condition; The first construction unit is further configured to construct a first three-dimensional rotation matrix based on the yaw angle, the pitch angle, and the roll angle of the first orthographic image; construct a first scaling matrix based on the elevation of the first orthographic image, wherein at least one of the yaw angle, the pitch angle, the roll angle, and the elevation is the image parameter to be determined; obtain an intrinsic parameter matrix of an image acquisition device, the image acquisition device being an image acquisition device that captured the first orthographic image and the second orthographic image; and construct a geographic coordinate function of a mapping geographic point to the first feature point based on the pixel coordinates of the first feature point in the feature point pair, the intrinsic parameter matrix, the first three-dimensional rotation matrix, and the first scaling matrix; The first construction unit is further used to construct a second three-dimensional rotation matrix based on the yaw angle, pitch angle and roll angle of the second orthographic image; construct a second scaling matrix based on the elevation of the second orthographic image, wherein at least one of the yaw angle, the pitch angle, the roll angle and the elevation is the image parameter to be determined; obtain an intrinsic parameter matrix of an image acquisition device, wherein the image acquisition device is an image acquisition device that takes the first orthographic image and the second orthographic image; and construct a geographic coordinate function of a mapping geographic point of the second feature point based on the intrinsic parameter matrix, the pixel coordinates of the second feature point in the feature point pair, the second three-dimensional rotation matrix and the second scaling matrix.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for determining image parameters according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to perform the method for determining image parameters according to any one of claims 1 to 7.

Images