An image target positioning method, system, electronic device and storage medium

Abstract

The present application relates to a kind of image target positioning method, system, electronic equipment and storage medium.The method includes the following steps: obtaining the pixel coordinate position of target in the image plane;The pixel coordinate position of target is converted into the world coordinate position in world coordinate system;In world coordinate system, obtain the ray with image shooting position as starting point, with the world coordinate position of starting point pointing to target as direction;Obtain the geographic field data of the place where image plane is located, obtain several three-dimensional vertex data sets containing elevation information and world coordinate position according to geographic field data and create three-dimensional grid according to three-dimensional vertex data set;The intersection of three-dimensional grid and ray is obtained, and the intersection position of three-dimensional grid and ray is judged;The elevation value of intersection point is obtained by interpolation using each three-dimensional vertex data of the three-dimensional grid where intersection point is located.In the present application, even in the region with complex terrain undulation, the position of target point can be efficiently and accurately obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image target positioning method, system, electronic equipment and storage medium. Background Art

[0002] The optoelectronic pod onboard a drone is capable of capturing real-time video of a target area, providing a timely response to the area's true situation. It is widely used in a variety of fields, including emergency response, disaster relief, and surveillance. After using the optoelectronic pod to capture video of the target area, it is often necessary to overlay each frame of the video with a map to match the video data with its spatial location. Therefore, in practical industry applications, it is necessary to quickly and accurately locate the target in the video image.

[0003] Existing technologies primarily employ two approaches. One involves image matching, which matches the real-time video captured by an optoelectronic pod with pre-stored images to determine the target's current location. This method requires pre-stored images and the matching of the currently captured image with the stored images. Therefore, this method places high demands on the computational efficiency of the real-time image matching algorithm, as well as the storage space and read / write speed.

[0004] The other type is geolocation, which uses the position and attitude information of the optoelectronic pod, the pixel position of the target in the image plane, and the ground height of the target to perform matrix conversion calculations to achieve the conversion between the image plane coordinates of the target and the geographic coordinates, and finally input the target image plane coordinates and output the target's geographic coordinates. However, when calculating the target height, this method assumes that the ground height of the target is an estimated value and that the height of the target area remains unchanged. Both conditions must be met, otherwise the target positioning accuracy cannot be guaranteed. However, in actual situations, the terrain height of the target area cannot remain completely unchanged, especially in complex terrain environments such as mountainous areas. Traditional geolocation methods often cannot accurately locate the target.

[0005] In addition, existing technologies have overcome the drawback of assuming the target's elevation is fixed by using iterative elevation. However, this method is not computationally efficient when the target's highest and lowest altitudes differ significantly. Each iteration requires retrieving the geographic field dataset and continuously iterating the elevation to obtain the deviation between the elevation corresponding to the latitude and longitude and the previous ground height until the deviation meets a threshold. This calculation method constantly reads and writes memory, resulting in excessive memory consumption and high spatial complexity. Furthermore, when the target's terrain is complex, this method cannot determine the iteration direction, leading to calculation errors. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to achieve efficient and accurate positioning of a target position in an image.

[0007] One of the technical solutions adopted by the present invention to solve the technical problem is: an image target positioning method, the method comprising the following steps:

[0008] Obtain the pixel coordinate position of the target in the image plane;

[0009] Converting the pixel coordinate position of the target into a world coordinate position in a world coordinate system;

[0010] In the world coordinate system, obtaining a ray starting from the image capturing position and pointing from the starting point to the world coordinate position of the target as a direction;

[0011] Acquiring geographic field data of a location of the image plane, obtaining a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions based on the geographic field data, and merging the three-dimensional vertex data sets to create a three-dimensional mesh;

[0012] Obtaining an intersection point between the three-dimensional grid and the ray, and determining a position of the intersection point between the three-dimensional grid and the ray;

[0013] If the intersection point is outside the three-dimensional grid, then calculating the intersection point of the next three-dimensional grid and the ray;

[0014] If the intersection point is located within the three-dimensional grid, obtaining the three-dimensional vertex data of the three-dimensional grid;

[0015] interpolating the three-dimensional vertex data of the three-dimensional grid where the intersection point is located to obtain the elevation value of the intersection point;

[0016] The world coordinate system position of the intersection point is converted into a geographic coordinate position in the geographic coordinate system and combined with the elevation value of the intersection point to obtain the geographic coordinate position of the target in the geographic coordinate system.

[0017] Furthermore, the obtaining of geographic field data of the location of the image plane comprises the following steps:

[0018] Obtaining pixel coordinate positions of four points on the image plane;

[0019] Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points;

[0020] The range of the image plane in the geographic coordinate system is determined according to the latitude and longitude, and geographic field data within the range is acquired.

[0021] Furthermore, the obtaining of geographic field data of the location of the image plane specifically includes the following steps:

[0022] Obtaining pixel coordinate positions of four points on the image plane;

[0023] Expanding the pixel coordinate positions of the four endpoints of the image plane toward the directions of the four endpoints;

[0024] Determine the pixel coordinate positions of the four points after expansion;

[0025] Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points after expansion;

[0026] The range of the image plane in the geographic coordinate system is determined according to the latitude and longitude, and geographic field data within the range is acquired.

[0027] Furthermore, the step of obtaining a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions according to the geographic field data and merging the three-dimensional vertex data sets to create a three-dimensional mesh according to the three-dimensional vertex data sets specifically includes the following steps:

[0028] Obtaining a geographic field data image containing elevation information and world coordinate positions based on the geographic field data within the range;

[0029] Acquire point data pixel by pixel in the geographic field data image to generate a three-dimensional vertex data set;

[0030] Creating an index array of a plurality of three-dimensional vertices as a group according to the three-dimensional vertex data set;

[0031] A three-dimensional grid is obtained after each group of index arrays is plotted in the image plane.

[0032] Furthermore, the number of three-dimensional vertices in the index array is three, and the three-dimensional grid is a triangular three-dimensional grid.

[0033] Furthermore, before obtaining the intersection point of the three-dimensional grid and the ray, the following steps are also included:

[0034] Dividing the image plane into a plurality of sub-regions, and creating a plurality of bounding boxes containing three-dimensional grids according to the sub-regions;

[0035] Obtaining the intersection of any of the bounding boxes and the ray, and determining the position of the intersection of the bounding box and the ray;

[0036] If the intersection point is outside the bounding box, then calculate the intersection point of the next bounding box and the ray;

[0037] If the intersection point is within the bounding box, all three-dimensional grids within the bounding box area are obtained.

[0038] Furthermore, the step of interpolating the three-dimensional vertex data of the three-dimensional grid where the intersection point is located to obtain the elevation value of the intersection point comprises the following steps:

[0039] Query the three 3D vertex data of the 3D grid according to the location of the intersection point;

[0040] The plane equation of the plane where the three-dimensional grid is located is:

[0041]

[0042] Where x and y are the coordinates of the intersection point, x1, y1, and z1 are the coordinates of one of the three-dimensional vertices A1, x2, y2, and z2 are the coordinates of the other three-dimensional vertex A2, and x3, y3, and z3 are the coordinates of the other three-dimensional vertex A3.

[0043] Substitute the coordinates of the intersection point into the following formula for interpolation to obtain the elevation value z of the intersection point;

[0044]

[0045] Among them, x 21 、y 21 、z 21 、x 31 、y 31 、y 31 is a variable defined by the following formula:

[0046] x 21 =x2-x1; x 31 =x3-x1

[0047] y 21 =y2-y1;y 31 =y3-y1

[0048] z 21 =z2-z1;z 31 =z3-z1

[0049] Among them, x 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the x direction, x 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the x direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the y direction, y 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the y direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the z direction, z 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the z direction.

[0050] Furthermore, the geographic field data of the location of the image plane includes a height map, and the elevation information is obtained by calculating the height value of the corresponding pixel position through the RGB value of each pixel in the height map.

[0051] Furthermore, the obtaining of the intersection of the three-dimensional grid and the ray and determining the position of the intersection of the three-dimensional grid and the ray comprises the following steps:

[0052] Get the 3D vertex data of any 3D mesh;

[0053] Convert the 3D vertex data of the current 3D mesh in the world coordinate system into a matrix type;

[0054] Obtaining normal data of the plane where the current three-dimensional grid is located, and determining whether there is an intersection between the ray and the plane based on whether the normal data is perpendicular to the ray;

[0055] If there is no intersection, obtain the 3D vertex data of the next 3D mesh;

[0056] If there is an intersection, the intersection position of the three-dimensional grid and the ray is determined.

[0057] Furthermore, the determining of the intersection position of the three-dimensional grid and the ray comprises the following steps:

[0058] Obtain edge segments formed between adjacent three-dimensional vertex data of the three-dimensional grid;

[0059] Obtaining the connecting line segments formed by connecting the intersection point and each three-dimensional vertex data;

[0060] Determine in turn whether each edge segment and the corresponding connecting segment intersect;

[0061] If not, the intersection point is outside the three-dimensional grid;

[0062] If so, the intersection point is located within the three-dimensional grid.

[0063] Another technical solution adopted by the present invention to solve the technical problem is: a positioning system for an image target, comprising:

[0064] A target position acquisition module is used to obtain the pixel coordinate position of the target in the image plane of the image;

[0065] A coordinate conversion module, configured to convert the pixel coordinate position of the target into a world coordinate position in a world coordinate system;

[0066] A ray acquisition module, configured to acquire, in the world coordinate system, a ray starting from the image capture position and pointing from the starting point to the world coordinate position of the target as a direction;

[0067] a three-dimensional mesh creation module, configured to obtain geographic field data of the location of the image plane, obtain a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions based on the geographic field data, and merge the three-dimensional vertex data sets to create a three-dimensional mesh;

[0068] An intersection determination module, configured to obtain an intersection of the three-dimensional grid and the ray, and determine a position of the intersection of the three-dimensional grid and the ray;

[0069] If the intersection point is outside the three-dimensional grid, then calculating the intersection point of the next three-dimensional grid and the ray;

[0070] If the intersection point is located within the three-dimensional grid, obtaining the three-dimensional vertex data of the three-dimensional grid;

[0071] an elevation value acquisition module, configured to interpolate the elevation value of the intersection point using the three-dimensional vertex data of the three-dimensional grid where the intersection point is located;

[0072] The target geographic coordinate position acquisition module converts the world coordinate system position of the intersection point into a geographic coordinate position in the geographic coordinate system and obtains the geographic coordinate position of the target in the geographic coordinate system by combining the elevation value of the intersection point.

[0073] Another technical solution adopted by the present invention to solve the technical problem is: an electronic device comprising:

[0074] memory for storing computer programs;

[0075] A processor is used to implement the steps of the above-mentioned image target positioning method when executing the computer program.

[0076] Another technical solution adopted by the present invention to solve its technical problem is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image target positioning method described above are implemented.

[0077] It can be seen from the above technical solutions that in this application, the image target positioning method obtains the elevation value of the target by calculating the intersection of the ray formed by the pixel point and the image shooting position with the three-dimensional grid, thereby obtaining the accurate positioning of the target. This method does not have pre-requirements such as the iteration direction and the initial elevation value setting, so it can greatly improve the efficiency of target positioning. In this application, the image target positioning method adopts a direct intersection method to obtain the position information of the target, and does not need to use elevation iteration to determine the target elevation value. Therefore, it does not cause the problem of excessive computational complexity and wrong iteration direction caused by elevation fluctuations, and can ensure that the target point position can be obtained efficiently and accurately even in areas with complex terrain fluctuations. In addition, the electronic device and computer-readable storage medium in this application can also achieve the above-mentioned effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention will be further described below with reference to the accompanying drawings and examples.

[0079] Figure 1 Flowchart of the image target positioning method in a specific embodiment of the present invention;

[0080] Figure 2 Flowchart of a method for acquiring geographic field data in a specific embodiment of the present invention;

[0081] Figure 3 Flowchart of a three-dimensional mesh creation method in a specific embodiment of the present invention;

[0082] Figure 4 Flowchart of the method for determining the intersection position in a specific embodiment of the present invention;

[0083] Figure 5 Flowchart of a method for determining whether an intersection point is located within a three-dimensional grid in a specific embodiment of the present invention;

[0084] Figure 6 Flowchart of a positioning method for obtaining a target position of an optoelectronic pod image by combining geographic field data in a specific embodiment of the present invention;

[0085] Figure 7 Schematic diagram of the intersection of rays and the image plane in a specific embodiment of the present invention;

[0086] Figure 8 A schematic diagram of the position between the intersection point and the three-dimensional grid in a specific embodiment of the present invention;

[0087] Figure 9 is a height map within the image plane in a specific embodiment of the present invention;

[0088] Figure 10 It is a three-dimensional grid map of the DEM for local visualization of the image plane in a specific embodiment of the present invention; DETAILED DESCRIPTION

[0089] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0090] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. It is obvious that the embodiment described is only one of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, other embodiments promoted by ordinary technicians in this field all fall within the scope of protection of the present invention.

[0091] The following is a brief explanation of the relevant abbreviations and key term definitions involved in the present invention.

[0092] OpenGL (Open Graphics Library) is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. This API consists of nearly 350 different function calls, used to draw everything from simple graphical bits to complex three-dimensional scenes. Another API system is Direct3D, which is available only on Microsoft Windows. OpenGL is commonly used in CAD, virtual reality, scientific visualization programs, and video game development.

[0093] A triangulated irregular network (TIN) uses a series of connected triangles to fit the earth's surface or other irregular surfaces. It is often used to construct digital terrain models, particularly digital elevation models. The most common method for generating these models is the Delaunay mesh. TINs offer many advantages for representing complex surfaces and are widely used in digital mapping, surface modeling and analysis, and in LIS.

[0094] Field Data Model: Field data is a special data type consisting of cells organized by coordinates or topology, each containing one or more attribute values. Field data measures continuous space, either in real life or through software simulations. Common measurements include temperature, velocity, and density. Field data is often referred to as spatial data, spatiotemporal data, or geospatial data because it is often related to space, time, or geographic location.

[0095] A Digital Elevation Model (DEM) is a digital simulation of the terrain (i.e., a digital representation of the terrain surface morphology) achieved through limited terrain elevation data. It is a physical ground model that represents ground elevation in the form of an ordered array of numerical values. It is a branch of the Digital Terrain Model (DTM), from which various other terrain characteristic values ​​can be derived. It is generally believed that a DTM describes the spatial distribution of linear and nonlinear combinations of various geomorphological factors, including elevation, such as slope, aspect, and slope change rate. Among them, a DEM is a zero-order, simple, single-item digital geomorphological model. Other geomorphological characteristics such as slope, aspect, and slope change rate can be derived based on the DEM.

[0096] A ray is a straight line formed by extending one end of a line segment infinitely. A ray has only one endpoint and its length cannot be measured. In Euclidean geometry, a shape formed by a point on a line and the part adjacent to it is called a ray.

[0097] This application provides the following embodiment: Figure 1 As shown, an image target positioning method includes the following steps, wherein it should be noted that there is no strict sequence relationship between some of the following steps, and some steps can be executed synchronously or in a preset order. The order of the steps in this application is only for the convenience of explanation and understanding.

[0098] S01. Obtain the pixel coordinate position of the target in the image plane of the image; wherein the image of the target can be obtained by directly shooting through a device such as a camera or an optoelectronic pod of a drone, or by capturing any frame from a recorded video.

[0099] S02: Convert the pixel coordinate position of the target into a world coordinate position in a world coordinate system.

[0100] S03. In the world coordinate system, obtain a ray starting from the image capture position and pointing from the start point to the world coordinate position of the target as a direction; wherein, taking the use of an optoelectronic pod for shooting as an example, the image capture position is the position of the optoelectronic pod at the moment of current image capture.

[0101] S04. Obtain geographic field data of the location of the image plane, obtain a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions according to the geographic field data, and merge the three-dimensional vertex data sets to create a three-dimensional grid.

[0102] The geographic field data of the location of the image plane includes a height map, and the elevation information is obtained by calculating the height value of the corresponding pixel position by the RGB value of each pixel in the height map.

[0103] Among them, such as Figure 2 As shown, in step S04, obtaining geographic field data of the location of the image plane includes the following steps:

[0104] S041, obtaining pixel coordinate positions of four points on the image plane;

[0105] S042. Obtaining the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points;

[0106] S043. Determine the range of the image plane in the geographic coordinate system according to the latitude and longitude, and obtain geographic field data within the range.

[0107] In certain embodiments, as Figure 3 As shown, in step S04, obtaining geographic field data of the location of the image plane may specifically include the following steps:

[0108] S041, obtaining pixel coordinate positions of four points on the image plane;

[0109] S042, expanding the pixel coordinate positions of the four endpoints of the image plane toward the directions of the four endpoints;

[0110] S043, determining the pixel coordinate positions of the four points after expansion;

[0111] S044. Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the expanded pixel coordinate positions of the four points;

[0112] S045. Determine the range of the image plane in the geographic coordinate system according to the latitude and longitude, and obtain geographic field data within the range.

[0113] The four points refer to the upper left, lower left, upper right, and lower right positions in the image plane. By obtaining the pixel coordinates of the upper left, lower left, upper right, and lower right points of the image plane and the longitude and latitude of the geographical coordinate system where these four coordinates are located, the approximate range of the geographical coordinate system of the image plane can be determined. The four-point expansion method is used to obtain a certain amount of redundant 3D vertex data, thereby ensuring that the position of objects at the edge of the image plane can be calculated.

[0114] The pixel coordinate positions of the four endpoints of the image plane obtained in step S041 are used to determine the positions and orientations of the four endpoints of the image, thereby enabling the four-boundary expansion of the image in step S042 to have a clear expansion orientation, and thus to obtain the pixel coordinate positions of the four endpoints after expansion. In addition, some of the four endpoints after expansion in step S043 may be the same as the four endpoints obtained in step S041, that is, the expansion amount of the image in one or more of the four directions is zero. In this case, the pixel coordinate positions of the four endpoints after expansion in that direction are the same as the pixel coordinate positions of the four endpoints obtained in step S041 in that direction.

[0115] Among them, such as Figure 3 As shown, in step S04, a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions are obtained according to the geographic field data and merged to create a three-dimensional mesh according to the three-dimensional vertex data sets, which specifically includes the following steps:

[0116] S046. Obtaining a geographic field data image containing elevation information and world coordinate position based on the geographic field data within the range;

[0117] S047, acquiring point data pixel by pixel in the geographic field data image to generate a three-dimensional vertex data set;

[0118] S048. Creating an index array with three three-dimensional vertices as a group according to the three-dimensional vertex data set;

[0119] S049. Draw each group of index arrays in the image plane to obtain a triangular three-dimensional grid.

[0120] A series of connected triangles in the image plane form a three-dimensional grid, which can form an irregular triangulated network (TIN) on the image plane, thereby fitting the surface within the image plane.

[0121] S05: Obtain an intersection point between the three-dimensional grid and the ray, and determine a position of the intersection point between the three-dimensional grid and the ray.

[0122] Among them, such as Figure 4 As shown, in step S05, obtaining the intersection of the three-dimensional grid and the ray and determining the position of the intersection of the three-dimensional grid and the ray include the following steps:

[0123] S051. Obtain three-dimensional vertex data of any three-dimensional grid;

[0124] S052. Convert the three-dimensional vertex data of the current three-dimensional grid in the world coordinate system into a matrix type;

[0125] S053, obtaining normal data of the plane where the current three-dimensional grid is located, and determining whether there is an intersection between the ray and the plane based on whether the normal data is perpendicular to the ray;

[0126] S054. If no intersection exists, obtain the three-dimensional vertex data of the next three-dimensional mesh;

[0127] S055. If an intersection exists, determine the intersection position of the three-dimensional grid and the ray.

[0128] Among them, such as Figure 5 As shown, in step S055, determining the intersection position of the three-dimensional grid and the ray includes the following steps:

[0129] S0551. Obtain edge segments formed between adjacent three-dimensional vertex data of the three-dimensional mesh;

[0130] S0552. Obtain connecting line segments formed by connecting the intersection point and each three-dimensional vertex data;

[0131] S0553. Determine in sequence whether each edge line segment intersects with the corresponding connecting line segment;

[0132] S0554: If not, the intersection point is located outside the three-dimensional grid;

[0133] S0555. If yes, the intersection point is located within the three-dimensional grid.

[0134] S06. If the intersection point is outside the three-dimensional grid, calculate the intersection point of the next three-dimensional grid and the ray;

[0135] S07. If the intersection point is located within the three-dimensional grid, obtaining three-dimensional vertex data of the three-dimensional grid;

[0136] S08, interpolating the three-dimensional vertex data of the three-dimensional grid where the intersection point is located to obtain the elevation value of the intersection point;

[0137] Wherein, in step S08, interpolating the elevation value of the intersection point using the three-dimensional vertex data of the three-dimensional grid where the intersection point is located includes the following steps:

[0138] S081. Query the three 3D vertex data of the 3D grid according to the location of the intersection point;

[0139] The plane equation of the plane where the three-dimensional grid is located is:

[0140]

[0141] Where x and y are the coordinates of the intersection point, x1, y1, and z1 are the coordinates of one of the three-dimensional vertices A1, x2, y2, and z2 are the coordinates of the other three-dimensional vertex A2, and x3, y3, and z3 are the coordinates of the other three-dimensional vertex A3.

[0142] S082. Substitute the coordinates of the intersection point into the following formula for interpolation to obtain the elevation value z of the intersection point;

[0143]

[0144] Among them, x 21 、y 21 、z 21 、x 31 、y 31 、y 31 is a variable defined by the following formula:

[0145] x 21 =x2-x1; x 31 =x3-x1

[0146] y 21 =y2-y1;y 31 =y3-y1

[0147] z 21 =z2-z1;z 31 =z3-z1

[0148] Among them, x 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the x direction, x 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the x direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the y direction, y 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the y direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the z direction, z 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the z direction.

[0149] S09: Convert the world coordinate system position of the intersection point into a geographic coordinate position in the geographic coordinate system and combine the elevation value of the intersection point to obtain the geographic coordinate position of the target in the geographic coordinate system.

[0150] In some embodiments, the method further includes the following steps between step S04 of creating a three-dimensional grid and step S05 of obtaining the intersection of the three-dimensional grid and the ray:

[0151] S10, dividing the image plane into a plurality of sub-regions, and creating a plurality of bounding boxes containing three-dimensional grids according to the sub-regions;

[0152] S11, obtaining an intersection point between any of the bounding boxes and the ray, and determining a position of the intersection point between the bounding box and the ray;

[0153] S12. If the intersection point is outside the bounding box, calculate the intersection point of the next bounding box and the ray;

[0154] S13. If the intersection point is within the bounding box, obtain all three-dimensional grids within the bounding box area.

[0155] Among them, this application will divide the image plane according to the size of the surface coverage of the image taken by the pod (that is, the geographical range covered by the image determined by the altitude and field of view of the pod), and the threshold range of the number of three-dimensional grids determined by the performance of the computer equipment running the program, and create several bounding boxes based on this. It will first determine whether the ray intersects with the bounding box and then input the triangle inside the bounding box for intersection calculation. This approach can greatly reduce the time complexity and space complexity of the calculation by reducing the number of calculations.

[0156] The present application also provides the following embodiment, a positioning method for obtaining the target position of an optoelectronic pod image in combination with geographic field data, wherein the method uses the boundaries of the area covered by the current frame image in the geographic coordinate system to obtain spatial three-dimensional data (i.e., a vertex data set containing X, Y, and Z) within the area from the geographic field data, creates a three-dimensional grid, and creates a ray from the geographic location of the optoelectronic pod to the geographic location of the target through the position and attitude information of the airborne optoelectronic pod and the pixel position of the target in the imaging plane. After determining the intersection of the ray and the three-dimensional grid, the accurate position of the target in the geographic coordinate system is obtained by using triangulation interpolation.

[0157] The steps of the positioning method are shown as follows: Figure 6 As shown, the specific steps of the positioning method are as follows:

[0158] S1: Get the pixel coordinate position of the target in the image plane (Xim, Yim)

[0159] x im =(x ip +o y )S y

[0160] y im =(y ip +o x )S x

[0161] Among them, xip and yip are the image element positions of the target in the image coordinate system, Ox and Oy are the pixel positions of the midpoint of the image, and Sx and Sy are the dimensions of each pixel in the xy direction;

[0162] S2: Convert the pixel coordinate position of the target in the image plane to the world coordinate position (XE, YE, ZE) in the world coordinate system:

[0163] Among them, the world coordinate system is the absolute coordinate system of the objective three-dimensional world, also known as the objective coordinate system. For a point P in space E =(X E ,Y E ,Z E ), the coordinates (XC, YC, ZC) of the corresponding camera (i.e. optoelectronic pod) coordinate system are related to the camera's attitude angle and GPS (location). During the flight of the drone, its attitude angle information can be obtained in real time through the IMU. In order to correct the camera posture.

[0164]

[0165] Among them, XE, YE and ZE are the positions of the target's image plane coordinates in the world coordinate system, and u and v are the pixel positions of the target in the image plane coordinate system (i.e., x in step S1). im 、y im ), u0, v0 is the center point in the image pixel coordinate system, that is, the pixel point corresponding to the image physical coordinate system (i.e., o in step S1 x 、o y ). f is the focal length of the pod photography.

[0166]

[0167] Matrix K is the pod's internal parameter matrix, containing four parameters related to the camera's internal structure. dx and dy represent the physical dimensions of the pixel on the X and Y axes, respectively, which are related to the camera's focal length, f.

[0168] is the external parameter matrix of the pod. Represent the yaw angle, pitch angle, and roll angle of the camera respectively.

[0169] S3: If Figure 7 As shown, a ray is created with the position of the pod in the world coordinate system as the ray starting point O and the ray direction D pointing from the pod position to the target position.

[0170] Wherein, plane P is the image plane, plane normal n is the normal perpendicular to plane P, and the distance from the origin of the world coordinate system to plane P is d;

[0171] The starting point of the ray is the position of the pod in the world coordinate system (i.e. the image shooting position), and the direction of the ray is from the position of the pod to the target position.

[0172] S4: Acquire a three-dimensional vertex data set within the coverage range of the pod image in the geographic coordinate system in the geographic field data and create a three-dimensional grid based on the three-dimensional vertex data set:

[0173] Among them, the specific method for determining the coverage range of the pod image in the geographical coordinate system can refer to the method disclosed in the invention patent with publication number CN109725340B. According to the four pixel coordinates of the upper left, lower left, upper right, and lower right of the pod image plane, the longitude and latitude of the geographical coordinate system where the four coordinates are located are obtained, thereby determining the approximate range of the geographical coordinate system where the image plane is located. In addition, in order to ensure that the position of the target at the edge of the photo can be calculated, a certain amount of redundant three-dimensional vertex data can be obtained by four-to-four expansion. Based on this range coordinate, the DEM data of the specified range can be cropped, and the data format is PNG, that is, the height map of the specified range is obtained, and the height value of the corresponding pixel position can be calculated based on the RGB value of each pixel in the height map. Through the three-dimensional space grid algorithm, the real three-dimensional terrain can be reconstructed based on the height map.

[0174] The steps for reconstructing the real 3D terrain based on the height map are as follows:

[0175] S41: Get the DEM data within the image plane coverage, such as Figure 8 The following shows the DEM data obtained by direct geolocation using the image plane's four coordinates:

[0176] S42: Starting from the upper left corner of the DEM image, point data is acquired pixel by pixel to generate three-dimensional vertex data.

[0177] S43: To reduce repeated drawing of points during drawing, an index array is created based on every three 3D vertices as a 3D grid, and a 3D grid is formed.

[0178] like Figure 9 The figure shows a local visualized DEM three-dimensional grid. Visualization may not be involved in actual calculations.

[0179] S5: Divide the three-dimensional grid set into a bounding box, and intersect the bounding box with the ray constructed by the positions of the pod and the target.

[0180] Based on computer performance parameters (number of GPU cores, GPU memory size), the optimal number of 3D meshes to be transferred to the GPU at a time is set. Using this number as a threshold, the overall 3D mesh surface within the surface area covered by the pod image (i.e., the geographic area covered by the image boundaries determined by the pod's altitude and field of view) is split into multiple sub-regions. Based on these sub-regions, AABB bounding boxes are created. The approximate location of the intersection of the ray and the 3D mesh is determined by calculating the intersection of the ray and the bounding box. This reduces the actual amount of computation, saves computational costs, and improves program efficiency. Using the OpenGL library, the intersection of the ray and the bounding box is calculated. If an intersection exists, the bounding box number is recorded and the next step is moved to. If no intersection exists, the loop continues with the next bounding box until all bounding boxes are calculated.

[0181] S6: Calculate the intersection of the ray constructed by the positions of the pod and the target with the three-dimensional grid.

[0182] Create a three-dimensional grid using the three-dimensional vertex data set obtained in a loop. First, convert the coordinates of each vertex of the three-dimensional vertex data set from geographic coordinates to world coordinates, and create the corresponding three-dimensional mesh (grid) through the OpenGL library. Calculate the intersection of the ray and the three-dimensional grid, obtain the location of the intersection, and store the location p(x, y) of the intersection. Use programmable pipeline technology to perform intersection operations on the created three-dimensional grid. The calculation logic is: input the three-dimensional mesh of the triangle into the calculation pipeline, first determine the spatial plane where the input three-dimensional mesh is located, then calculate the intersection of the ray and the plane, and determine whether the intersection falls inside the three-dimensional mesh of the triangle. If not, jump to the next input. If the intersection falls inside the three-dimensional mesh plane of the triangle, store the intersection position and the vertex data of this three-dimensional mesh.

[0183] In step S5, the logic for calculating the intersection of the ray and the plane where the three-dimensional grid is located and whether the intersection is inside the three-dimensional grid is as follows:

[0184] First, convert the vertex arrays of the three-dimensional mesh of the triangle in the world coordinate system into matrix type;

[0185] Calculate the normal data of the plane where the current input 3D mesh is located;

[0186] Determine whether the ray intersects the plane (excluding the case where the ray direction is parallel to the plane), and obtain the distance from the intersection point to the starting point of the ray;

[0187] Determine whether the intersection point is inside the 3D grid, such as Figure 8As shown, if two line segments intersect, then the two points on either side of one line segment must lie on either side of the other line segment. To determine whether two line segments intersect, we determine the relationship between the intersection point and the 3D mesh: We sequentially determine whether the three sides AC, CB, and BA of the 3D mesh triangle intersect with the line segments BD, AD, and CD formed by the intersection point and the vertices of the opposite sides of the 3D mesh triangle, and determine whether the intersection point is inside the 3D mesh triangle. As shown in the figure, the intersection point D and the vertices of the 3D mesh triangle do not intersect with the opposite sides, indicating that point D is inside the 3D mesh triangle. The line segment formed by the intersection point E and vertex B intersects with the side AC opposite point B, so points E and F are not inside the 3D mesh triangle.

[0188] S7: Convert the world coordinate system position of the intersection point to the geographic coordinate system position

[0189] Query the 3D grid based on the location of the intersection point p(x,y), obtain the coordinates of the three 3D vertices A1(x1,y1,z1), A2(x2,y2,z2), and A3(x3,y3,z3) of the 3D grid, and use the following formula for interpolation to obtain the elevation value of the intersection point;

[0190]

[0191] Among them, x1, y1, z1, x2, y2, z2, x3, y3, z3 are the vertex coordinates of the three-dimensional mesh of the triangle, x 21 、y 21 、z 21 、x 31 、y 31 、y 31 is a variable defined by the following formula:

[0192] x 21 =x2-x1; x 31 =x3-x1

[0193] y 21 =y2-y1;y 31 =y3-y1

[0194] z 21 =z2-z1;z 31 =z3-z1

[0195] Among them, x 21 is the difference between the vertices A2 and A1 of the triangle in the x direction, x 31 is the difference between the vertices A3 and A1 of the triangle in the x direction, 21 is the difference between the vertices A2 and A1 of the triangle in the y direction, y 31 is the difference between the vertices A3 and A1 of the triangle in the y direction, 21is the difference between the vertices A2 and A1 of the triangle in the z direction, z 31 is the difference between the vertices A3 and A1 of the triangle in the z direction.

[0196] Based on the above method for creating a three-dimensional grid based on geographic data to obtain an electro-optical pod's image target positioning, the verification process and results are presented for a real-world case study using the aircraft's position, attitude, pod attitude, and FOV (field of view). The aircraft's GPS longitude, latitude, and altitude are: 104.491, 31.6271, and 1239.21, respectively. The aircraft's attitude angles are: -25.1127, -1.65585, and 0.0114592 degrees, respectively. The pod's attitude angles are: 0.00572958, -89.9945, and 0 degrees, respectively. The pod's FOV is: 0.5639 degrees. With the above position, angle, and other information fixed, the target point in the acquired real-time video feed, located to the upper right of the center of the image, is selected. The pixel coordinates of the target point are: 980, 565. After locating the image target using the positioning method above, the final calculated target location longitude, latitude, and altitude are 104.484, 31.637, and 629.56, respectively, which are the same as the target's actual geographic location.

[0197] The positioning method of the photoelectric pod image target obtained by creating a three-dimensional grid in combination with geographic field data in this application can meet the requirements of fast and high-precision target positioning; in the calculation process, the corresponding vertex data set containing the height is obtained from the geographic field data by calculating the four ranges of the image plane, and a three-dimensional grid is constructed. According to the creation ray of the pod position and the target position, the intersection of the ray and the plane where the three-dimensional grid is located is calculated, and the position of the intersection and the three-dimensional grid surrounding it are obtained. Then, the elevation value of the target point is obtained by elevation interpolation. The position and elevation values ​​obtained in this calculation process are accurate values, so they can meet the application scenarios with high requirements for position and elevation. The data source required to construct the three-dimensional grid in this application has wide scalability, that is, data that meets the characteristics of geographic field data can be used to create a three-dimensional grid, that is, all spatial data that meets continuous coverage and can obtain X, Y, Z attributes only need to be expanded in the acquisition function of the vertex data set, and the rest of the process remains unchanged.

[0198] The present application also provides the following embodiment, a system for positioning an image target, comprising:

[0199] A target position acquisition module is used to obtain the pixel coordinate position of the target in the image plane of the image;

[0200] A coordinate conversion module, configured to convert the pixel coordinate position of the target into a world coordinate position in a world coordinate system;

[0201] A ray acquisition module, configured to acquire, in the world coordinate system, a ray starting from the image capture position and pointing from the starting point to the world coordinate position of the target as a direction;

[0202] a three-dimensional mesh creation module, configured to obtain geographic field data of the location of the image plane, obtain a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions based on the geographic field data, and merge the three-dimensional vertex data sets to create a three-dimensional mesh;

[0203] An intersection determination module, configured to obtain an intersection of the three-dimensional grid and the ray, and determine a position of the intersection of the three-dimensional grid and the ray;

[0204] If the intersection point is outside the three-dimensional grid, then calculating the intersection point of the next three-dimensional grid and the ray;

[0205] If the intersection point is located within the three-dimensional grid, obtaining the three-dimensional vertex data of the three-dimensional grid;

[0206] an elevation value acquisition module, configured to interpolate the elevation value of the intersection point using the three-dimensional vertex data of the three-dimensional grid where the intersection point is located;

[0207] The target geographic coordinate position acquisition module converts the world coordinate system position of the intersection point into a geographic coordinate position in the geographic coordinate system and obtains the geographic coordinate position of the target in the geographic coordinate system by combining the elevation value of the intersection point.

[0208] The working principle and positioning method of the image target positioning system of the present application can refer to the above-mentioned image target positioning method or the positioning method for obtaining the image target position of the optoelectronic pod in combination with geographic field data, and will not be repeated here.

[0209] The present application also provides the following embodiment, an electronic device, comprising:

[0210] memory for storing computer programs;

[0211] The processor is used to implement the steps of the above-mentioned image target positioning method or the above-mentioned positioning method for obtaining the image target position of the optoelectronic pod in combination with geographic field data when executing the computer program.

[0212] The present application also provides the following embodiment: a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned image target positioning method or the above-mentioned positioning method for obtaining the image target position of an optoelectronic pod in combination with geographic field data.

[0213] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0214] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0215] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image target positioning method, characterized in that: include: Obtain the pixel coordinate position of the target in the image plane; Converting the pixel coordinate position of the target into a world coordinate position in a world coordinate system; In the world coordinate system, obtaining a ray starting from the image capturing position and pointing from the starting point to the world coordinate position of the target as a direction; Acquiring geographic field data of a location of the image plane, obtaining a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions based on the geographic field data, and merging the three-dimensional vertex data sets to create a three-dimensional mesh; Obtaining an intersection point between the three-dimensional grid and the ray, and determining a position of the intersection point between the three-dimensional grid and the ray; If the intersection point is outside the three-dimensional grid, then calculating the intersection point of the next three-dimensional grid and the ray; If the intersection point is located within the three-dimensional grid, obtaining the three-dimensional vertex data of the three-dimensional grid; interpolating the three-dimensional vertex data of the three-dimensional grid where the intersection point is located to obtain the elevation value of the intersection point; Converting the world coordinate system position of the intersection point into a geographic coordinate position in the geographic coordinate system and combining the elevation value of the intersection point to obtain the geographic coordinate position of the target in the geographic coordinate system; The step of obtaining geographic field data of the location of the image plane comprises the following steps: Obtaining pixel coordinate positions of four points on the image plane; Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points; Determining a range of the image plane in a geographic coordinate system according to the latitude and longitude, and acquiring geographic field data within the range; The steps of obtaining a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions according to the geographic field data and merging the three-dimensional vertex data sets to create a three-dimensional mesh according to the three-dimensional vertex data sets specifically include the following steps: Obtaining a geographic field data image containing elevation information and world coordinate positions based on the geographic field data within the range; Acquire point data pixel by pixel in the geographic field data image to generate a three-dimensional vertex data set; Creating an index array of a plurality of three-dimensional vertices as a group according to the three-dimensional vertex data set; A three-dimensional grid is obtained after each group of index arrays is plotted in the image plane.

2. The image target positioning method according to claim 1, characterized in that: The step of obtaining geographic field data of the location of the image plane specifically includes the following steps: Obtaining pixel coordinate positions of four points on the image plane; Expanding the pixel coordinate positions of the four endpoints of the image plane toward the directions of the four endpoints; Determine the pixel coordinate positions of the four points after expansion; Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points after expansion; The range of the image plane in the geographic coordinate system is determined according to the latitude and longitude, and geographic field data within the range is acquired.

3. The image target positioning method according to claim 1, characterized in that: The number of three-dimensional vertices in the index array is three, and the three-dimensional mesh is a triangular three-dimensional mesh.

4. The image target positioning method according to claim 1, characterized in that: Before obtaining the intersection point of the three-dimensional grid and the ray, the following steps are also included: Dividing the image plane into a plurality of sub-regions, and creating a plurality of bounding boxes containing three-dimensional grids according to the sub-regions; Obtaining the intersection of any of the bounding boxes and the ray, and determining the position of the intersection of the bounding box and the ray; If the intersection point is outside the bounding box, then calculate the intersection point of the next bounding box and the ray; If the intersection point is within the bounding box, all three-dimensional grids within the bounding box area are obtained.

5. The image target positioning method according to claim 4, characterized in that: The method of interpolating the elevation value of the intersection point by using the three-dimensional vertex data of the three-dimensional grid where the intersection point is located comprises the following steps: Query the three 3D vertex data of the 3D grid according to the location of the intersection point; The plane equation of the plane where the three-dimensional grid is located is: Where x and y are the coordinates of the intersection point, x1, y1, and z1 are the coordinates of one of the three-dimensional vertices A1, x2, y2, and z2 are the coordinates of the other three-dimensional vertex A2, and x3, y3, and z3 are the coordinates of the other three-dimensional vertex A3. Substitute the coordinates of the intersection point into the following formula for interpolation to obtain the elevation value z of the intersection point; Among them, x 21 、y 21 、z 21 、x 31 、y 31 、y 31 is a variable defined by the following formula: x 21 =x2-x1;x 31 =x3-x1 y 21 =y2-y1;y 31 =y3-y1 With 21 =z2-z1;z 31 =z3-z1 Among them, x 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the x direction, x 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the x direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the y direction, y 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the y direction, 21 is the difference between the three-dimensional vertex A2 and the three-dimensional vertex A1 in the z direction, z 31 is the difference between the three-dimensional vertex A3 and the three-dimensional vertex A1 in the z direction.

6. The image target positioning method according to claim 1, characterized in that: The geographic field data of the location of the image plane includes a height map, and the elevation information is obtained by calculating the height value of the corresponding pixel position by the RGB value of each pixel in the height map.

7. The image target positioning method according to claim 1, characterized in that: The obtaining of the intersection of the three-dimensional grid and the ray and determining the position of the intersection of the three-dimensional grid and the ray comprises the following steps: Get the 3D vertex data of any 3D mesh; Convert the 3D vertex data of the current 3D mesh in the world coordinate system into a matrix type; Obtaining normal data of the plane where the current three-dimensional grid is located, and determining whether there is an intersection between the ray and the plane based on whether the normal data is perpendicular to the ray; If there is no intersection, obtain the 3D vertex data of the next 3D mesh; If there is an intersection, the intersection position of the three-dimensional grid and the ray is determined.

8. The image target positioning method according to claim 7, characterized in that: Determining the intersection position of the three-dimensional grid and the ray comprises the following steps: Obtain edge segments formed between adjacent three-dimensional vertex data of the three-dimensional grid; Obtaining the connecting line segments formed by connecting the intersection point and each three-dimensional vertex data; Determine in turn whether each edge segment and the corresponding connecting segment intersect; If not, the intersection point is outside the three-dimensional grid; If so, the intersection point is located within the three-dimensional grid.

9. A positioning system for an image target, characterized in that: include: A target position acquisition module is used to obtain the pixel coordinate position of the target in the image plane of the image; A coordinate conversion module, configured to convert the pixel coordinate position of the target into a world coordinate position in a world coordinate system; A ray acquisition module, configured to acquire, in the world coordinate system, a ray starting from the image capture position and pointing from the starting point to the world coordinate position of the target as a direction; a three-dimensional mesh creation module, configured to obtain geographic field data of the location of the image plane, obtain a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions based on the geographic field data, and merge the three-dimensional vertex data sets to create a three-dimensional mesh; An intersection determination module, configured to obtain an intersection of the three-dimensional grid and the ray, and determine a position of the intersection of the three-dimensional grid and the ray; If the intersection point is outside the three-dimensional grid, then calculating the intersection point of the next three-dimensional grid and the ray; If the intersection point is located within the three-dimensional grid, obtaining the three-dimensional vertex data of the three-dimensional grid; an elevation value acquisition module, configured to interpolate the elevation value of the intersection point using the three-dimensional vertex data of the three-dimensional grid where the intersection point is located; The target geographic coordinate position acquisition module converts the world coordinate system position of the intersection point into the geographic coordinate position in the geographic coordinate system and combines the elevation value of the intersection point to obtain the geographic coordinate position of the target in the geographic coordinate system; The step of obtaining geographic field data of the location of the image plane comprises the following steps: Obtaining pixel coordinate positions of four points on the image plane; Obtain the longitude and latitude of the four pixel coordinate positions in the geographic coordinate system through the pixel coordinate positions of the four points; Determining a range of the image plane in a geographic coordinate system according to the latitude and longitude, and acquiring geographic field data within the range; The steps of obtaining a plurality of three-dimensional vertex data sets containing elevation information and world coordinate positions according to the geographic field data and merging the three-dimensional vertex data sets to create a three-dimensional mesh according to the three-dimensional vertex data sets specifically include the following steps: Obtaining a geographic field data image containing elevation information and world coordinate positions based on the geographic field data within the range; Acquire point data pixel by pixel in the geographic field data image to generate a three-dimensional vertex data set; Creating an index array of a plurality of three-dimensional vertices as a group according to the three-dimensional vertex data set; A three-dimensional grid is obtained after each group of index arrays is plotted in the image plane.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the image target positioning method according to any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image target positioning method according to any one of claims 1 to 9.

Images