A map overlay method, apparatus, device and medium

Abstract

The application relates to the technical field of image processing, in particular to a map superimposition method and device, equipment and medium. In the embodiment of the application, the electronic device determines a target homography matrix for perspective transformation of the orthographic image and the video frame according to the position information of the target pixel point pairs belonging to the same entity in the orthographic image and the video frame, so that the second position information of the target to be added in the orthographic image can be determined according to the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the orthographic image, and then the third position information of the first position information in the video frame can be determined according to the target homography matrix. When the map is superimposed, the target to be added is superimposed on the video frame shot by the unmanned aerial vehicle through a pure visual calculation method, the attitude information of the unmanned aerial vehicle is not relied on, and the accuracy of the map superimposition is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a map overlay method, apparatus, device and medium. Background Technology

[0002] Currently, with the continuous development of information technology, drones are playing an important role in various fields. In real-world scenarios, drones are frequently used in inspection, such as inspecting protected farmland areas to detect any damage. In practical applications, a region containing geographic information is usually planned in advance, and map overlay technology is used to overlay this region onto the video frames captured by the drone. This marks the region in the video frames, improving the human-computer interaction experience and facilitating real-time decision-making and subsequent detection and judgment.

[0003] Existing technologies for map overlay typically rely on the drone's attitude information to calculate the coordinates of the area to be overlaid in the coordinate system corresponding to the video frame, and then mark the area in the video frame based on the calculated coordinates. However, since drones are moving platforms, attitude information errors are unavoidable, significantly impacting the final geographic information overlay effect. Summary of the Invention

[0004] This application provides a map overlay method to solve the problems in the prior art where attitude information errors of UAVs are unavoidable, resulting in poor map overlay effects and low accuracy when map overlay is based on attitude information.

[0005] In a first aspect, embodiments of this application provide a map overlay method, the method comprising:

[0006] The saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed are input into the pre-trained matching model to obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0007] Based on the position information of the target pixel pairs and the pre-configured homography matrix algorithm, a target homography matrix is ​​determined for perspective transformation of the orthophoto image and the video frame;

[0008] Obtain the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-saved orthophoto image, as well as the first position information of the target to be added in the geodetic coordinate system; determine the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, and determine the third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0009] Secondly, embodiments of this application also provide a map overlay device, the device comprising:

[0010] The matching module is used to input the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into the pre-trained matching model, and obtain the position information of the target pixel point pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0011] The processing module is configured to determine, based on the position information of the target pixel pairs and a pre-configured homography matrix algorithm, a target homography matrix for perspective transformation of the orthophoto image and the video frame; acquire pre-saved affine transformation parameters of the image coordinate system and the geodetic coordinate system of the orthophoto image and a first position information of the target to be added in the geodetic coordinate system; determine a second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information; and determine a third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0012] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the map overlay method described above.

[0013] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the map overlay method described above.

[0014] In this embodiment, the electronic device inputs a stored orthophoto image containing the region corresponding to the video frame to be overlaid, and the video frame to be overlaid, into a pre-trained matching model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model; based on the position information of the target pixel pairs and a pre-configured homography matrix algorithm, a target homography matrix for perspective transformation of the orthophoto image and the video frame is determined; the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-stored orthophoto image and the first position information of the target to be added in the geodetic coordinate system are obtained; based on the affine transformation parameters and the first position information, the second position information of the target to be added in the orthophoto image is determined; and based on the target homography matrix and the second position information, the third position information of the target to be added in the video frame is determined. In this embodiment, the electronic device determines the target homography matrix for perspective transformation of the orthophoto image and video frame by using the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame. This allows the second position information of the target to be added in the orthophoto image to be determined first based on the affine transformation parameters between the image coordinate system and the geodetic coordinate system of the orthophoto image. Then, based on the target homography matrix, the third position information of the first position information in the video frame is determined. In this embodiment, when performing map overlay, the target to be added is overlaid onto the video frame captured by the UAV using a pure visual calculation method, without relying on the attitude information of the UAV, thus improving the accuracy of map overlay. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a map overlay process provided in an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of the corner points of a video frame provided in an embodiment of this application;

[0018] Figure 3 A schematic diagram of the superposition of line segments provided in an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of the overlay of a first type of region provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of the overlay of the second type of region provided in the embodiments of this application;

[0021] Figure 6 This is a schematic diagram of the overlay of a third region provided in an embodiment of this application;

[0022] Figure 7a This is a first type of overlay diagram including video frame corner points provided in an embodiment of this application;

[0023] Figure 7b This is a second overlay diagram containing video frame corner points provided in an embodiment of this application;

[0024] Figure 7c This is a third type of overlay diagram containing video frame corner points provided in an embodiment of this application;

[0025] Figure 7d This is a fourth type of overlay diagram including video frame corner points provided in the embodiments of this application;

[0026] Figure 8 This is a schematic diagram of the map overlay process provided in an embodiment of this application;

[0027] Figure 9a This is a schematic diagram of satellite imagery or map overlay provided in an embodiment of this application;

[0028] Figure 9b This is a schematic diagram illustrating the overlay of video frames from a drone, provided in an embodiment of this application.

[0029] Figure 10 This is a schematic diagram of the structure of a map overlay device provided in an embodiment of this application;

[0030] Figure 11 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] To improve the accuracy of map overlay, embodiments of this application provide a map overlay method, apparatus, device, and medium.

[0033] In this embodiment, the electronic device inputs a stored orthophoto image containing the region corresponding to the video frame to be overlaid, and the video frame to be overlaid, into a pre-trained matching model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model; based on the position information of the target pixel pairs and a pre-configured homography matrix algorithm, a target homography matrix for perspective transformation of the orthophoto image and the video frame is determined; the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-stored orthophoto image and the first position information of the target to be added in the geodetic coordinate system are obtained; based on the affine transformation parameters and the first position information, the second position information of the target to be added in the orthophoto image is determined; and based on the target homography matrix and the second position information, the third position information of the target to be added in the video frame is determined.

[0034] Example 1:

[0035] Figure 1 This application provides a schematic diagram of a map overlay process, which includes:

[0036] S101: Input the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into the pre-trained matching model, and obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0037] The map overlay method provided in this application is applied to an electronic device, which may be a PC or a server.

[0038] This application proposes a map overlay method. The input image is an orthophoto of a UAV, the first position of the target to be added in the geodetic coordinate system, and a video frame captured by the UAV. The target to be added is overlaid onto the video frame using a pure visual calculation method, without relying on the accuracy of the UAV's attitude.

[0039] Specifically, in this embodiment, the electronic device stores an orthophoto of an unmanned person and a video frame to be overlaid. The orthophoto includes the region corresponding to the video frame, and can be obtained by stitching together multiple orthophoto sub-images taken by a drone. For example, pix4d software can be used for stitching.

[0040] In this embodiment, after determining the orthophoto image and the video frame, the electronic device performs image registration on the orthophoto image and the video frame, enabling subsequent overlay of the video frame based on the orthophoto image. Specifically, the electronic device uses a neural network to extract features from the orthophoto image and the video frame respectively, and performs matching and filtering on the extracted pixels to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and the video frame. This position information consists of the coordinates of the pixel in the orthophoto image and the coordinates of the other pixel in the video frame.

[0041] Specifically, in this embodiment, the electronic device inputs an orthophoto image and a video frame into a pre-trained matching model. The matching model determines the feature value corresponding to each pixel in the orthophoto image and video frame, and based on the feature value, determines target pixel pairs belonging to the same entity in the orthophoto image and video frame, and outputs the position information of the target pixel pairs in the orthophoto image and video frame. The electronic device obtains the position information of the target pixel pairs output by the matching model in the orthophoto image and video frame.

[0042] In this embodiment, since the location information output by the matching model consists of the coordinates of two pixels in different images, the order of the coordinates in the output location information can be predetermined during the training of the matching model. For example, the first coordinate in the location information can be specified as the coordinate of the pixel located in the orthophoto image, and the second coordinate as the coordinate of the pixel located in the video frame; or the first coordinate in the location information can be specified as the coordinate of the pixel located in the video frame, and the second coordinate as the coordinate of the pixel located in the orthophoto image. The order of the coordinates is not restricted here. The electronic device also stores the order of the coordinates in the location information, which allows for better parsing of the location information output by the matching model later.

[0043] For example, the position information of the target pixel pair output by the electronic device can be [(20,40),(55,80)]. The order of the coordinates pre-saved in the electronic device is that the first coordinate in the position information is the coordinate of the pixel in the orthophoto image, and the second coordinate is the coordinate of the pixel in the video frame. Then the electronic device determines that the coordinate of the pixel in the orthophoto image of the target pixel pair is (20,40), and the coordinate of the pixel in the video frame of the target pixel pair is (55,80).

[0044] S102: Based on the position information of the target pixel pairs in the orthophoto image and the video frame, and a pre-configured homography matrix algorithm, determine the target homography matrix for perspective transformation of the orthophoto image and the video frame.

[0045] In this embodiment of the application, after determining the target pixel pairs belonging to the same entity in the orthophoto image and video frame, the electronic device can determine the target homography matrix for perspective transformation of the orthophoto image and video frame based on the position information of the target pixel pairs in the orthophoto image and video frame and a pre-configured homography matrix algorithm.

[0046] In the embodiments of this application, the perspective transformation of pixels in an orthophoto image to a video frame, or the perspective transformation of pixels in a video frame to an orthophoto image, is achieved through the target homography matrix.

[0047] In this embodiment of the application, the process by which the electronic device determines the target homography matrix based on the target similarity point pairs is prior art and will not be described in detail here.

[0048] S103: Obtain the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-saved orthophoto image, as well as the first position information of the target to be added in the geodetic coordinate system; determine the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, and determine the third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0049] In this embodiment of the application, the orthophoto image is an ortho-projected image. The orthophoto image carries geographic information, that is, the electronic device can determine and save the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the orthophoto image based on the orthophoto image.

[0050] Based on this, in this embodiment, when there is a target to be added to a video frame, the electronic device can determine the second position information of the target in the orthophoto image based on the first position information of the target in the geodetic coordinate system and the affine transformation parameters corresponding to the orthophoto image. Then, based on the determined target homography matrix used to perform perspective transformation on the orthophoto image and the video frame, the electronic device determines the third position information of the second position information in the video frame, and superimposes the target onto the third position information, thus achieving the superposition of the video frames.

[0051] Specifically, in this embodiment, the affine transformation parameters corresponding to the orthophoto image include at least: the east-west geographic coordinates x of the top-left corner of the orthophoto image, the east-west resolution, and the east-west rotation parameters; the north-south geographic coordinates y of the top-left corner of the orthophoto image, the north-south resolution, and the north-south rotation parameters. Here, the resolution is the actual distance between two adjacent pixels in the north-south or east-west direction.

[0052] In this embodiment of the application, the electronic device can determine the second position information in the orthophoto image of the target to be added using the following formula:

[0053]

[0054]

[0055] Where (x, y) is the first location information of the target to be added, (px, py) is the second location information of the target to be added, geotrans[0] is the east-west geographic coordinate x of the upper left corner of the orthophoto image, geotrans[1] is the east-west resolution of the orthophoto image, geotrans[2] is the east-west rotation parameter of the orthophoto image, geotrans[3] is the north-south geographic coordinate y of the upper left corner of the orthophoto image, geotrans[4] is the north-south rotation parameter of the orthophoto image, and geotrans[5] is the north-south resolution of the orthophoto image.

[0056] In this embodiment, the process by which the electronic device determines the third position information of the target to be added in the video frame based on the target homography matrix and the second position information is prior art and will not be described in detail here.

[0057] In this embodiment of the application, the first location information is the coordinates of the target to be added in the geodetic coordinate system, the second location information is the coordinates of the target to be added in the orthophoto image, and the third location information is the coordinates of the target to be added in the video frame.

[0058] In this embodiment, the electronic device determines the target homography matrix for perspective transformation of the orthophoto image and video frame by using the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame. This allows the second position information of the target to be added in the orthophoto image to be determined first based on the affine transformation parameters between the image coordinate system and the geodetic coordinate system of the orthophoto image. Then, based on the target homography matrix, the third position information of the first position information in the video frame is determined. In this embodiment, when performing map overlay, the target to be added is overlaid onto the video frame captured by the UAV using a pure visual calculation method, without relying on the attitude information of the UAV, thus improving the accuracy of map overlay.

[0059] Example 2:

[0060] To improve the accuracy of map overlay, based on the above embodiments, in this embodiment of the application, the matching model includes: a feature extraction sub-model and a matching sub-model;

[0061] The step of inputting the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into a pre-trained matching model, and obtaining the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model, includes:

[0062] The orthophoto image and the video frame are input into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel.

[0063] The orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel are input into the matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

[0064] In this embodiment, the matching model includes a feature extraction sub-model and a matching sub-model. The feature extraction sub-model is used for feature extraction, determining the feature value corresponding to each pixel. The matching sub-model is used to determine target pixel pairs belonging to the same entity based on the feature value corresponding to each pixel, and outputs the position information of the target pixel pairs.

[0065] Specifically, in this embodiment, the electronic device inputs orthophotos and video frames into a feature extraction sub-model to obtain orthophotos and video frames output by the feature extraction sub-model, each labeled with the feature value corresponding to a pixel. The electronic device then inputs the orthophotos and video frames labeled with the feature value corresponding to each pixel into a matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophotos and video frames output by the matching sub-model.

[0066] Example 3:

[0067] To improve the accuracy of map overlay, based on the above embodiments, in this embodiment, after obtaining the position information of target pixel pairs belonging to the same entity in the orthophoto image output by the matching model and the video frame, the method further includes:

[0068] For each pair of target pixels, the fundamental matrix corresponding to the target pixel is determined based on the pre-saved eight-point algorithm and other pairs of target pixels.

[0069] For each base matrix, a target ray is determined based on the base matrix and one pixel in each target pixel pair, and the target distance from the other pixel in the target pixel pair to the target ray is determined; if the target distance does not exceed a preset distance threshold, the target pixel pair is determined to be the pixel pair to be retained corresponding to the base matrix;

[0070] Determine the basis matrix that has the largest number of corresponding pixel pairs to be retained, and update the target pixel pairs using the pixel pairs to be retained corresponding to the determined basis matrix.

[0071] In this embodiment of the application, after the electronic device determines the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame through the matching model, the electronic device will filter the target pixel pairs in order to further improve the accuracy of map overlay.

[0072] Specifically, in the embodiments of this application, since video frame imaging is perspective imaging, it can usually be represented by a fundamental matrix with orthophoto. Therefore, the fundamental matrix is ​​solved using the RANSAC-based eight-point method, and the target pixel points are filtered during the solution process.

[0073] Specifically, in the embodiments of this application, the electronic device determines the basic matrix corresponding to each target pixel point for each target pixel point pair based on the pre-saved eight-point algorithm and the position information of every other seven target pixel point pairs and the position information of the target pixel point pair.

[0074] For each base matrix, the electronic device determines a target ray based on the base matrix and one pixel in each target pixel pair, and determines the target distance from the other pixel in the target pixel pair to the target ray; if the target distance does not exceed a preset distance threshold, the target pixel pair is determined to be the pixel pair to be retained corresponding to the base matrix.

[0075] The electronic device determines the basic matrix with the largest number of corresponding pixel pairs to be retained, and identifies the pixel pairs to be retained corresponding to this basic matrix as the filtered target pixel pairs.

[0076] Example 4:

[0077] To improve the accuracy of map overlay, based on the above embodiments, in this embodiment, the method for determining the third position information of the target to be added in the video frame based on the target homography matrix and the second position information further includes:

[0078] Using the k-nearest neighbor algorithm and the second location information, a preset number of candidate target pixels belonging to the target pixel pair are determined in the orthophoto image that are adjacent to the target to be added.

[0079] The target homography matrix is ​​updated based on the candidate target pixel pairs corresponding to the candidate target pixels and the homography matrix algorithm.

[0080] In this embodiment, since the target homography matrix is ​​a global homography matrix determined based on all target pixel pairs, considering that the scenes in the UAV image capture area are usually not on the same plane, directly using the global homography matrix cannot accurately describe the relationship between the video frame and the orthophoto. Therefore, the electronic device calculates the local homography matrix based on k target pixels near the target to be detected. This local homography matrix can better describe the transformation relationship between the two images of the location of the target to be added.

[0081] Specifically, in this embodiment, the electronic device determines a preset number of candidate target pixels belonging to target pixel pairs that are adjacent to the target in the orthophoto image, based on the second position information of the target to be added in the orthophoto image; wherein the preset number is k, and k is a positive integer. The electronic device updates the target homography matrix based on the candidate target pixel pairs corresponding to the candidate target pixels and the homography matrix algorithm, and determines the third position information of the target to be added in the video frame based on the updated target homography matrix.

[0082] Considering that the target to be added may be a group of points, performing L2 distance calculations on all target pixels for each target would be computationally intensive. Therefore, a k-nearest neighbor algorithm can be used to construct a KD search tree, which is then used to determine candidate target pixels. Based on this, except for the first target's k-nearest neighbor search, which requires building a KD search tree, the efficiency of subsequent geographic information point searches is greatly improved.

[0083] Example 5:

[0084] To improve the accuracy of map overlay, based on the above embodiments, in this embodiment, after determining the second position information of the target to be added in the orthophoto image according to the affine transformation parameters and the first position information, the method further includes:

[0085] Based on the target homography matrix and the fourth position information of the corner points of the video frame, the target region of the video frame in the orthophoto image is determined;

[0086] Determine whether the second location information is within the target area; if so, proceed to the next step of determining the third location information of the target to be added in the video frame based on the target homography matrix and the second location information.

[0087] In order to improve the efficiency of map overlay and avoid overlaying maps on targets not captured in video frames, in this embodiment of the application, the electronic device first determines whether the target to be added is in the video frame before determining the second location information of the target in the video frame.

[0088] In this embodiment, the electronic device obtains the homography transformation from the video frame to the orthophoto image through target homography matrix estimation, and then obtains the corresponding coordinates of the four corner points of the orthophoto image on the orthophoto image. Since the orthophoto image stitched by pix4d contains 6 affine transformation parameters, which describe the relationship between the image coordinate system and the geodetic coordinate system of the orthophoto image, the first position information of the target to be added can be converted into the corresponding second position information on the orthophoto image. Then, it is determined whether the second position information of the target to be added on the orthophoto image is within the target area formed by the four corner points on the orthophoto image, and thus whether the target point appears in the UAV image.

[0089] Specifically, in this embodiment of the application, after determining the target homography matrix and the second position information of the target to be added in the orthophoto image, the electronic device determines the target area of ​​the video frame in the orthophoto image based on the target homography matrix and the fourth position information of the four corner points of the video frame.

[0090] The electronic device determines whether the second location information of the target to be added is within the target area. If so, the electronic device performs the subsequent step of determining the third location information of the target to be added in the video frame based on the target homography matrix and the second location information.

[0091] In this embodiment of the application, the electronic device can determine whether the second location information is within the target area using the following formula:

[0092] Angle p1p_p1p2 =(x p2 -x p1 )*(y p -y p1 )-(x p -x p1 )*(y p2 -y p1 )

[0093] Angle p2p_p2p3 =(x p3 -x p2 )*(y p -y p2 )-(x p -x p2 )*(y p3 -y p2 )

[0094] Angle p3p_p3p4 =(x p4 -x p3 )*(y p -y p3 )-(x p -x p3 )*(y p4 -y p3 )

[0095] Angle p4p_p4p1 =(x p1 -x p4 )*(y p -y p4 )-(x p -x p4 )*(y p -y p4 )

[0096] Where p is the target to be added, p1, p2, p3, and p4 are the four corner points sorted clockwise, and Angle p1p_p1p2 Let p1p be the angle between the vector p1p formed by the target point p and the corner point p1, and the corner point vector p1p2. Similarly, Angle p2p_p2p3 Let Angle be the angle between vectors p2p and p2p3. p3p_p3p4 Let Angle be the angle between vectors p3p and p3p4. p4p_p4p1 Let p4p be the angle between vectors p4p and p4p1. If all four angles are less than 0 or greater than 0, then the target p to be added is considered to be in the UAV imagery; otherwise, the target p to be added is considered not to be in the UAV imagery.

[0097] Among them, the corner points of a video frame are the four vertices of the video frame, and the position information of the corner points of a video frame is the coordinates of the vertices of the video frame in the image coordinate system corresponding to the video frame.

[0098] Figure 2 This is a schematic diagram of the corner points of a video frame provided in an embodiment of this application. Figure 2 As shown in the figure, the rectangle represents a video frame, and vertices A, B, C, and D of the rectangle are the corner points of the video frame.

[0099] Example 6:

[0100] To improve the accuracy of map overlay, based on the above embodiments, in this embodiment of the application, if the second location information is not within the target area, the method further includes:

[0101] Determine whether the target to be added is an endpoint of a pre-saved line segment to be added;

[0102] If so, then obtain the fifth position information of the other endpoint of the line segment to be added in the orthophoto image;

[0103] Based on the fifth location information and the second location information, determine the first intersection point between the line connecting the other endpoint and the target to be added and the target area;

[0104] Based on the target homography matrix and the position information corresponding to the first intersection point with the smallest distance to the target to be added, the third position information of the target to be added in the video frame is determined.

[0105] In this embodiment, the target to be added can be a single point or an endpoint of a line segment. If the target to be added is a single point, and the electronic device determines that the second location information of the target to be added is not within the target area corresponding to the video frame, then the electronic device will not perform map overlay processing on the target to be added.

[0106] Furthermore, if the target to be added is an endpoint of the line segment to be added, the electronic device will further determine whether there are any sub-segments of the line segment to be added within the target area corresponding to the video frame.

[0107] Specifically, in this embodiment, when the electronic device determines that the second location information is not within the target area, the electronic device determines whether the target to be added is an endpoint of a pre-saved line segment to be added. If the electronic device determines that the target to be added is an endpoint of the line segment to be added, the electronic device obtains the fifth location information of the other endpoint of the saved line segment to be added in the orthophoto image, and determines the first intersection point of the line connecting the other endpoint and the target to be added with the target area based on the fifth location information and the second location information; and determines the third location information of the target to be added in the video frame based on the target homography matrix and the location information corresponding to the first intersection point with the smallest distance to the target to be added.

[0108] That is, in the embodiments of this application, the electronic device superimposes the sub-line segment portion of the line segment to be added in the video frame where the line segment appears in the video frame.

[0109] Figure 3 This is a schematic diagram of the superposition of line segments provided in the embodiments of this application, as shown in the figure. Figure 3 As shown, region A is the target region corresponding to the video frame in the orthophoto image, and point B is the target to be added. Point B is an endpoint of line segment BC. Based on the second position information of point B in the orthophoto image, the electronic device determines that point B is not within the target region. Therefore, the electronic device determines the intersection point D of line segment BC and the target region, and uses the position information of point D in the video frame as the third position information of point B in the video frame, then overlays it onto the video frame. The map overlay at this point is the overlay of line segment CD onto the video frame.

[0110] Furthermore, in this embodiment of the application, when overlaying a map onto a region or line segment, the target homography matrix can be used to determine whether there are points between two consecutive geographic information points contained in the target that intersect with the edge of the video frame. If one or both of the two consecutive target points are not within the target area corresponding to the video frame, it is determined whether the line connecting the two points intersects with the target area; if so, the coordinates of the intersection point are output.

[0111] Among them, the fifth position information of the other endpoint of the line segment to be added in the orthophoto image is the coordinate of the other endpoint in the orthophoto image.

[0112] Example 7:

[0113] To improve the accuracy of map overlay, based on the above embodiments, the method in this application embodiment further includes:

[0114] If the target to be added is not an endpoint of a pre-saved line segment to be added, then determine whether the target to be added is a corner point of a pre-saved area to be added;

[0115] If so, then determine the first corner point and the second corner point adjacent to the target to be added in the area to be added; and obtain the saved sixth position information of the first corner point in the orthophoto image and the seventh position information of the second corner point in the orthophoto image;

[0116] Based on the sixth position information and the second position information, determine the second intersection point of the line connecting the first corner point and the target to be added with the target area, and based on the seventh position information and the second position information, determine the third intersection point of the line connecting the second corner point and the target to be added with the target area.

[0117] Based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance to the target to be added, the first sub-position information of the target to be added in the video frame is determined;

[0118] Based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance to the target to be added, the second sub-position information of the target to be added in the video frame is determined.

[0119] In this embodiment, the target to be added can be a single point or a corner of the area to be added. If the target to be added is a single point, and the electronic device determines that the second location information of the target to be added is not within the target area corresponding to the video frame, then the electronic device will not perform map overlay processing on the target to be added.

[0120] Furthermore, if the target to be added is a corner point of the area to be added, the electronic device will further determine whether there is a sub-region of the area to be added within the target area corresponding to the video frame.

[0121] Specifically, in this embodiment, when the electronic device determines that the second location information is not within the target area, and determines that the target to be added is not an endpoint of a pre-saved line segment to be added, the electronic device determines whether the target to be added is a corner point of a pre-saved area to be added. If the electronic device determines that the target to be added is a corner point of the area to be added, the electronic device determines the first corner point and the second corner point adjacent to the target to be added in the area to be added; and obtains the sixth location information of the first corner point in the orthophoto image and the seventh location information of the second corner point in the orthophoto image.

[0122] The electronic device determines the second intersection point of the line connecting the first corner point and the target to be added with the target area based on the sixth and second position information, and determines the third intersection point of the line connecting the second corner point and the target to be added with the boundary of the target area based on the seventh and second position information. The electronic device determines the first sub-position information of the target to be added in the video frame based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance to the target; and determines the second sub-position information of the target to be added in the video frame based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance to the target.

[0123] That is, in the embodiments of this application, the electronic device superimposes the sub-region of the video frame in which the line segment to be added appears.

[0124] Figure 4 This is a schematic diagram of the overlay of a first type of region provided in an embodiment of this application, as shown below. Figure 4 As shown, region A is the target region corresponding to the video frame in the orthophoto image, and point B is the target to be added. Point B is an endpoint of region BCDE. Based on the second position information of point B in the orthophoto image, the electronic device determines that point B is not within the target region. Therefore, the electronic device determines the first corner point C and the second corner point E adjacent to point B in region BCDE. The electronic device determines the intersection point M of line segment BC and the boundary of the target region, and the intersection point N of line segment BE and the boundary of the target region. It then uses the first and second sub-position information of intersection points N and M in the video frame as the third position information of point B in the video frame and overlays them. The resulting map overlay is region CDEMN overlaid in the video frame.

[0125] In this embodiment, the sixth position information of the first corner point in the orthophoto image is the coordinate of the first corner point in the orthophoto image, and the seventh position information of the second corner point in the orthophoto image is the coordinate of the second corner point in the orthophoto image.

[0126] In addition, the following situations may also occur in the embodiments of this application:

[0127] Case 1: Among the two corner points adjacent to the target to be added, the number of corner points where the line connecting the target to be added intersects with the target area of ​​the orthophoto image is 1.

[0128] If the above situation exists, the electronic device determines the position information of the intersection point in the video frame based on the target homography matrix and the position information of the intersection point in the orthophoto image, and determines the position information of the intersection point in the video frame as the third position information of the target to be added in the video frame.

[0129] Then, the electronic device determines the position of the area to be added in the video frame based on the third position information of each corner point of the area to be added in the video frame.

[0130] Figure 5 This is a schematic diagram of the overlay of a second type of region provided in an embodiment of this application, as shown below. Figure 5 As shown, region A is the target region corresponding to the video frame in the orthophoto image, and point B is the target to be added. Point B is an endpoint of region BCDE. Based on the second position information of point B in the orthophoto image, the electronic device determines that point B is not within the target region. Therefore, the electronic device determines the first corner point C and the second corner point E adjacent to point B in region BCDE. The electronic device determines that the intersection point M of line segment BC and the boundary of the target region, and that line segment BE and the boundary of the target region do not intersect, then the electronic device determines the position information of intersection point M in the video frame as the third position information of point B in the video frame. Using the same method, the electronic device determines the position information of intersection point N in the video frame as the third position information of point E in the video frame. In summary, the electronic device determines the superimposed region of the region to be added in the video frame as region CDNM.

[0131] Scenario 2: Among the two corner points adjacent to the target to be added, the number of corner points where the line connecting the target to be added intersects with the target area of ​​the orthophoto image is 0.

[0132] If situation two exists, the electronic device determines that the video frame does not contain the target to be added. Then, the electronic device overlays the video frame based on the third position information of other corner points of the area to be added in the video frame.

[0133] Figure 6 This is a schematic diagram of the overlay of a third region provided in an embodiment of this application, as shown below. Figure 6As shown, region A is the target region corresponding to the video frame in the orthophoto image, and point B is the target to be added. Point B is an endpoint of region BCDE. Based on the second position information of point B in the orthophoto image, the electronic device determines that point B is not within the target region. Therefore, the electronic device determines the first corner point C and the second corner point E adjacent to point B in region BCDE. The electronic device determines that line segment BC and line segment BE do not intersect with the boundary of the target region. Therefore, the electronic device determines that the position information of intersection point N in the video frame is the third position information of point E in the video frame, and the position information of intersection point M in the video frame is the third position information of point C in the video frame. In summary, the electronic device determines that the superimposed region of the region to be added in the video frame is region DMN.

[0134] Furthermore, in this embodiment, the area to be added may contain corner points of video frames. If this is the case, the electronic device will also overlay the video frames based on the corner points. Figures 7a-7d As shown, regions ABCD are the target regions corresponding to the video frames in the orthophoto image, and regions abcd are the regions to be added. Figure 7a This is a first type of overlay diagram including video frame corner points provided in an embodiment of this application. Figure 7a This indicates that the region ABMN is superimposed in the video frame; Figure 7b This is a second type of overlay diagram containing video frame corner points provided in an embodiment of this application. Figure 7b This represents the region ANcM overlaid in a video frame; Figure 7c This is a third type of overlay diagram including video frame corner points provided in the embodiments of this application. Figure 7c This indicates the region ABNMD overlaid in the video frame; Figure 7d This is a fourth type of overlay diagram including video frame corner points provided in the embodiments of this application. Figure 7d This indicates the overlay region ABCD in the video frame.

[0135] Furthermore, in this embodiment, the area to be added can be any shape other than a rectangle, and there are no restrictions on this. The superposition process for areas of other shapes is the same as the superposition process for areas of matrices.

[0136] Figure 8 This is a schematic diagram of the map overlay process provided in the embodiments of this application, as shown below. Figure 8 As shown, the process includes: image registration, basic matrix estimation, target homography matrix calculation, KD search tree construction, searching for k nearest neighbors, updating the target homography matrix based on the nearest neighbors, and obtaining third location information through homography transformation.

[0137] The map overlay method provided in this application can also be applied to overlay satellite images or maps. Figure 9aThis is a schematic diagram of satellite image overlay provided in an embodiment of this application. Figure 9b This is a schematic diagram illustrating the overlay of video frames from a drone as provided in an embodiment of this application. Figure 9a and Figure 9b As shown, when overlaying, the overlay can be an area, a point, or a line segment, etc.

[0138] This application embodiment can support the overlay of geographic information of point, line segment, and region types onto UAV video; this application embodiment calculates the local homography matrix based on the k nearest points of the target to be added, which can better reflect the transformation relationship between video frames and orthophotos in the current local area compared to the global homography matrix; this application embodiment uses the fundamental matrix estimation method to filter out some abnormally matched target pixel pairs, which can improve the accuracy of subsequent homography matrix estimation.

[0139] Example 8:

[0140] Based on the above embodiments, this application also provides a map overlay device. Figure 10 This is a schematic diagram of a map overlay device provided in an embodiment of this application. The device includes:

[0141] The matching module 1001 is used to input the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into the pre-trained matching model, and obtain the position information of the target pixel point pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0142] The processing module 1002 is configured to determine, based on the position information of the target pixel pairs and a pre-configured homography matrix algorithm, a target homography matrix for perspective transformation of the orthophoto image and the video frame; acquire, pre-saved, affine transformation parameters of the image coordinate system and the geodetic coordinate system of the orthophoto image and a first position information of the target to be added in the geodetic coordinate system; determine, based on the affine transformation parameters and the first position information, a second position information of the target to be added in the orthophoto image; and determine, based on the target homography matrix and the second position information, a third position information of the target to be added in the video frame.

[0143] In one possible implementation, the matching model includes: a feature extraction sub-model and a matching sub-model;

[0144] The matching module 1001 is specifically used to input the orthophoto image and the video frame into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel; and to input the orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel into the matching sub-model to obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

[0145] In one possible implementation, the processing module 1002 is further configured to: for each target pixel pair, determine the base matrix corresponding to the target pixel based on a pre-saved eight-point algorithm and other target pixel pairs; for each base matrix, determine a target ray based on the base matrix and one pixel in each target pixel pair, and determine the target distance from the other pixel in the target pixel pair to the target ray; if the target distance does not exceed a preset distance threshold, determine the target pixel pair as the pixel pair to be retained corresponding to the base matrix; determine the base matrix with the largest number of corresponding pixel pairs to be retained, and update the target pixel pair using the pixel pairs to be retained corresponding to the determined base matrix.

[0146] In one possible implementation, the processing module 1002 is further configured to use the k-nearest neighbor algorithm and the second position information to determine a preset number of candidate target pixels in the orthophoto image that are adjacent to the target to be added and belong to the target pixel pair; and to update the target homography matrix according to the candidate target pixel pair corresponding to the candidate target pixel and the homography matrix algorithm.

[0147] In one possible implementation, the processing module 1002 is further configured to determine the target region of the video frame in the orthophoto image based on the target homography matrix and the fourth position information of the corner points of the video frame; determine whether the second position information is within the target region; if so, perform the subsequent step of determining the third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0148] In one possible implementation, the processing module 1002 is further configured to determine whether the target to be added is an endpoint of a pre-saved line segment to be added; if so, to obtain the fifth position information of the other endpoint of the saved line segment to be added in the orthophoto image; to determine the first intersection point of the line connecting the other endpoint and the target to be added with the target area based on the fifth position information and the second position information; and to determine the third position information of the target to be added in the video frame based on the target homography matrix and the position information corresponding to the first intersection point with the smallest distance from the target to be added.

[0149] In one possible implementation, the processing module 1002 is further configured to: if the target to be added is not an endpoint of a pre-saved line segment to be added, determine whether the target to be added is a corner point of a pre-saved region to be added; if so, determine a first corner point and a second corner point adjacent to the target to be added in the region to be added; and obtain the sixth position information of the first corner point in the orthophoto image and the seventh position information of the second corner point in the orthophoto image; determine the second intersection point of the line connecting the first corner point and the target to be added with the target region based on the sixth position information and the second position information, and determine the third intersection point of the line connecting the second corner point and the target to be added with the target region based on the seventh position information and the second position information; determine the first sub-position information of the target to be added in the video frame based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance from the target to be added; and determine the second sub-position information of the target to be added in the video frame based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance from the target to be added.

[0150] Example 9:

[0151] Based on the above embodiments, Figure 11 This application provides a schematic diagram of an electronic device structure based on the above embodiments. The present application further provides an electronic device, such as... Figure 11 As shown, it includes: processor 1101, communication interface 1102, memory 1103 and communication bus 1104, wherein processor 1101, communication interface 1102 and memory 1103 communicate with each other through communication bus 1104.

[0152] The memory 1103 stores a computer program, which, when executed by the processor 1101, causes the processor 1101 to perform the following steps:

[0153] The saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed are input into the pre-trained matching model to obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0154] Based on the position information of the target pixel pairs and the pre-configured homography matrix algorithm, a target homography matrix is ​​determined for perspective transformation of the orthophoto image and the video frame;

[0155] Obtain the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-saved orthophoto image, as well as the first position information of the target to be added in the geodetic coordinate system; determine the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, and determine the third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0156] In one possible implementation, the matching model includes: a feature extraction sub-model and a matching sub-model;

[0157] The step of inputting the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into a pre-trained matching model, and obtaining the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model, includes:

[0158] The orthophoto image and the video frame are input into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel.

[0159] The orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel are input into the matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

[0160] In one possible implementation, after obtaining the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching model, the method further includes:

[0161] For each pair of target pixels, the fundamental matrix corresponding to the target pixel is determined based on the pre-saved eight-point algorithm and other pairs of target pixels.

[0162] For each base matrix, a target ray is determined based on the base matrix and one pixel in each target pixel pair, and the target distance from the other pixel in the target pixel pair to the target ray is determined; if the target distance does not exceed a preset distance threshold, the target pixel pair is determined to be the pixel pair to be retained corresponding to the base matrix;

[0163] Determine the basis matrix that has the largest number of corresponding pixel pairs to be retained, and update the target pixel pairs using the pixel pairs to be retained corresponding to the determined basis matrix.

[0164] In one possible implementation, before determining the third position information of the target to be added in the video frame based on the target homography matrix and the second position information, the method further includes:

[0165] Using the k-nearest neighbor algorithm and the second location information, a preset number of candidate target pixels belonging to the target pixel pair are determined in the orthophoto image that are adjacent to the target to be added.

[0166] The target homography matrix is ​​updated based on the candidate target pixel pairs corresponding to the candidate target pixels and the homography matrix algorithm.

[0167] In one possible implementation, after determining the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, the method further includes:

[0168] Based on the target homography matrix and the fourth position information of the corner points of the video frame, the target region of the video frame in the orthophoto image is determined;

[0169] Determine whether the second location information is within the target area; if so, proceed to the next step of determining the third location information of the target to be added in the video frame based on the target homography matrix and the second location information.

[0170] In one possible implementation, if the second location information is not within the target area, the method further includes:

[0171] Determine whether the target to be added is an endpoint of a pre-saved line segment to be added;

[0172] If so, then obtain the fifth position information of the other endpoint of the line segment to be added in the orthophoto image;

[0173] Based on the fifth location information and the second location information, determine the first intersection point between the line connecting the other endpoint and the target to be added and the target area;

[0174] Based on the target homography matrix and the position information corresponding to the first intersection point with the smallest distance to the target to be added, the third position information of the target to be added in the video frame is determined.

[0175] In one possible implementation, the method further includes:

[0176] If the target to be added is not an endpoint of a pre-saved line segment to be added, then determine whether the target to be added is a corner point of a pre-saved area to be added;

[0177] If so, then determine the first corner point and the second corner point adjacent to the target to be added in the area to be added; and obtain the saved sixth position information of the first corner point in the orthophoto image and the seventh position information of the second corner point in the orthophoto image;

[0178] Based on the sixth position information and the second position information, determine the second intersection point of the line connecting the first corner point and the target to be added with the target area, and based on the seventh position information and the second position information, determine the third intersection point of the line connecting the second corner point and the target to be added with the target area.

[0179] Based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance to the target to be added, the first sub-position information of the target to be added in the video frame is determined;

[0180] Based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance to the target to be added, the second sub-position information of the target to be added in the video frame is determined.

[0181] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the map overlay method, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated parts will not be described again.

[0182] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0183] Communication interface 1102 is used for communication between the above-mentioned electronic device and other devices.

[0184] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0185] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0186] Example 10:

[0187] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0188] The saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed are input into the pre-trained matching model to obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model.

[0189] Based on the position information of the target pixel pairs and the pre-configured homography matrix algorithm, a target homography matrix is ​​determined for perspective transformation of the orthophoto image and the video frame;

[0190] Obtain the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-saved orthophoto image, as well as the first position information of the target to be added in the geodetic coordinate system; determine the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, and determine the third position information of the target to be added in the video frame based on the target homography matrix and the second position information.

[0191] In one possible implementation, the matching model includes: a feature extraction sub-model and a matching sub-model;

[0192] The step of inputting the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into a pre-trained matching model, and obtaining the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model, includes:

[0193] The orthophoto image and the video frame are input into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel.

[0194] The orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel are input into the matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

[0195] In one possible implementation, after obtaining the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching model, the method further includes:

[0196] For each pair of target pixels, the fundamental matrix corresponding to the target pixel is determined based on the pre-saved eight-point algorithm and other pairs of target pixels.

[0197] For each base matrix, a target ray is determined based on the base matrix and one pixel in each target pixel pair, and the target distance from the other pixel in the target pixel pair to the target ray is determined; if the target distance does not exceed a preset distance threshold, the target pixel pair is determined to be the pixel pair to be retained corresponding to the base matrix;

[0198] Determine the basis matrix that has the largest number of corresponding pixel pairs to be retained, and update the target pixel pairs using the pixel pairs to be retained corresponding to the determined basis matrix.

[0199] In one possible implementation, before determining the third position information of the target to be added in the video frame based on the target homography matrix and the second position information, the method further includes:

[0200] Using the k-nearest neighbor algorithm and the second location information, a preset number of candidate target pixels belonging to the target pixel pair are determined in the orthophoto image that are adjacent to the target to be added.

[0201] The target homography matrix is ​​updated based on the candidate target pixel pairs corresponding to the candidate target pixels and the homography matrix algorithm.

[0202] In one possible implementation, after determining the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, the method further includes:

[0203] Based on the target homography matrix and the fourth position information of the corner points of the video frame, the target region of the video frame in the orthophoto image is determined;

[0204] Determine whether the second location information is within the target area; if so, proceed to the next step of determining the third location information of the target to be added in the video frame based on the target homography matrix and the second location information.

[0205] In one possible implementation, if the second location information is not within the target area, the method further includes:

[0206] Determine whether the target to be added is an endpoint of a pre-saved line segment to be added;

[0207] If so, then obtain the fifth position information of the other endpoint of the line segment to be added in the orthophoto image;

[0208] Based on the fifth location information and the second location information, determine the first intersection point between the line connecting the other endpoint and the target to be added and the target area;

[0209] Based on the target homography matrix and the position information corresponding to the first intersection point with the smallest distance to the target to be added, the third position information of the target to be added in the video frame is determined.

[0210] In one possible implementation, the method further includes:

[0211] If the target to be added is not an endpoint of a pre-saved line segment to be added, then determine whether the target to be added is a corner point of a pre-saved area to be added;

[0212] If so, then determine the first corner point and the second corner point adjacent to the target to be added in the area to be added; and obtain the saved sixth position information of the first corner point in the orthophoto image and the seventh position information of the second corner point in the orthophoto image;

[0213] Based on the sixth position information and the second position information, determine the second intersection point of the line connecting the first corner point and the target to be added with the target area, and based on the seventh position information and the second position information, determine the third intersection point of the line connecting the second corner point and the target to be added with the target area.

[0214] Based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance to the target to be added, the first sub-position information of the target to be added in the video frame is determined;

[0215] Based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance to the target to be added, the second sub-position information of the target to be added in the video frame is determined.

[0216] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the map overlay method, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.

[0217] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0218] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0219] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0220] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0221] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A map overlay method, characterized in that, The method includes: The saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed are input into the pre-trained matching model to obtain the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model. Based on the position information of the target pixel pairs and the pre-configured homography matrix algorithm, a target homography matrix is ​​determined for perspective transformation of the orthophoto image and the video frame; Obtain the affine transformation parameters of the image coordinate system and the geodetic coordinate system of the pre-saved orthophoto image, as well as the first position information of the target to be added in the geodetic coordinate system; determine the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, and determine the third position information of the target to be added in the video frame based on the target homography matrix and the second position information; The matching model includes a feature extraction sub-model and a matching sub-model. The step of inputting the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into a pre-trained matching model, and obtaining the position information of the target pixel pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model, includes: The orthophoto image and the video frame are input into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel. The orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel are input into the matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

2. The method according to claim 1, characterized in that, After obtaining the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching model, the method further includes: For each pair of target pixels, the fundamental matrix corresponding to the target pixel is determined based on the pre-saved eight-point algorithm and other pairs of target pixels. For each base matrix, a target ray is determined based on the base matrix and one pixel in each target pixel pair, and the target distance from the other pixel in the target pixel pair to the target ray is determined; if the target distance does not exceed a preset distance threshold, the target pixel pair is determined to be the pixel pair to be retained corresponding to the base matrix; Determine the base matrix that has the largest number of corresponding pixel pairs to be retained, and update the target pixel pairs using the pixel pairs to be retained corresponding to the determined base matrix.

3. The method according to claim 1, characterized in that, The method of determining, based on the target homography matrix and the second position information, that the target to be added is located before the third position information in the video frame, further includes: Using the k-nearest neighbor algorithm and the second location information, a preset number of candidate target pixels belonging to the target pixel pair are determined in the orthophoto image that are adjacent to the target to be added. The target homography matrix is ​​updated based on the candidate target pixel pairs corresponding to the candidate target pixels and the homography matrix algorithm.

4. The method according to claim 1, characterized in that, After determining the second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information, the method further includes: Based on the target homography matrix and the fourth position information of the corner points of the video frame, the target region of the video frame in the orthophoto image is determined; Determine whether the second location information is within the target area; if so, proceed to the next step of determining the third location information of the target to be added in the video frame based on the target homography matrix and the second location information.

5. The method according to claim 4, characterized in that, If the second location information is not within the target area, the method further includes: Determine whether the target to be added is an endpoint of a pre-saved line segment to be added; If so, then obtain the fifth position information of the other endpoint of the line segment to be added in the orthophoto image; Based on the fifth location information and the second location information, determine the first intersection point between the line connecting the other endpoint and the target to be added and the target area; Based on the target homography matrix and the position information corresponding to the first intersection point with the smallest distance to the target to be added, the third position information of the target to be added in the video frame is determined.

6. The method according to claim 5, characterized in that, The method further includes: If the target to be added is not an endpoint of a pre-saved line segment to be added, then determine whether the target to be added is a corner point of a pre-saved area to be added; If so, then determine the first corner point and the second corner point adjacent to the target to be added in the area to be added; and obtain the saved sixth position information of the first corner point in the orthophoto image and the seventh position information of the second corner point in the orthophoto image; Based on the sixth position information and the second position information, determine the second intersection point of the line connecting the first corner point and the target to be added with the target area, and based on the seventh position information and the second position information, determine the third intersection point of the line connecting the second corner point and the target to be added with the target area. Based on the target homography matrix and the position information corresponding to the second intersection point with the smallest distance to the target to be added, the first sub-position information of the target to be added in the video frame is determined; Based on the target homography matrix and the position information corresponding to the third intersection point with the smallest distance to the target to be added, the second sub-position information of the target to be added in the video frame is determined.

7. A map overlay device, characterized in that, The device includes: The matching module is used to input the saved orthophoto image containing the region corresponding to the video frame to be superimposed and the video frame to be superimposed into the pre-trained matching model, and obtain the position information of the target pixel point pairs belonging to the same entity in the orthophoto image and the video frame output by the matching model. The processing module is configured to: determine a target homography matrix for perspective transformation of the orthophoto image and the video frame based on the position information of the target pixel pairs and a pre-configured homography matrix algorithm; acquire pre-saved affine transformation parameters of the image coordinate system and the geodetic coordinate system of the orthophoto image and the first position information of the target to be added in the geodetic coordinate system; determine a second position information of the target to be added in the orthophoto image based on the affine transformation parameters and the first position information; and determine a third position information of the target to be added in the video frame based on the target homography matrix and the second position information. The matching model includes a feature extraction sub-model and a matching sub-model. The matching module is specifically used to input the orthophoto image and the video frame into the feature extraction sub-model to obtain the orthophoto image and video frame output by the feature extraction sub-model, which are labeled with the feature values ​​corresponding to each pixel; and to input the orthophoto image and video frame labeled with the feature values ​​corresponding to each pixel into the matching sub-model to obtain the position information of target pixel pairs belonging to the same entity in the orthophoto image and video frame output by the matching sub-model.

8. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being used to implement the steps of the map overlay method as described in any one of claims 1-6 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the map overlay method as described in any one of claims 1-6.

Images