Target positioning method and related apparatus

Abstract

The application discloses a target positioning method and related device, comprising: acquiring a first image and a second image, the first image and the second image being obtained by shooting a same target object at a first position and a second position respectively; and acquiring a distance between the first position and the second position obtained by a ranging device; performing feature extraction and matching on the first image and the second image respectively to obtain a first feature point pair set; obtaining a first rotation matrix and a first translation vector according to the first feature point pair set; and determining direction information and distance of the target object according to the first translation vector, the first rotation matrix, coordinates of the target object in the first image and the second image, and the distance between the first position and the second position. The application also discloses a positioning device. The target positioning method of the application is used for target positioning, and a positioning result with better positioning precision can be obtained, and the position of the target positioning method is not exposed.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a target localization method and related apparatus. Background Technology

[0002] Obtaining the absolute position of a target on the battlefield not only enables drones to conduct precision strikes but also serves as navigation and positioning information for other weapon platforms. For a soldier on the battlefield, obtaining the precise position of a target involves calculating the target's azimuth and distance, and then combining this with their own coordinates and orientation to arrive at the target's accurate location. Most existing target location methods are based on binocular or multi-view vision systems, calculating the target's position by solving the limiting constraints between multiple cameras. Alternatively, adding a laser rangefinder to the vision system can more easily provide distance information between the target and the soldier, allowing them to combine this with the line-of-sight to determine the target's position.

[0003] On the battlefield, the enemy uses laser detectors to determine the location of scouts. While a target positioning system that combines a laser rangefinder and a camera can achieve good positioning accuracy, it is also easy for counter-reconnaissance to expose one's own position. Summary of the Invention

[0004] This application provides a target positioning method and related apparatus. Using the target positioning method of this application, a positioning result with good positioning accuracy can be obtained, while not revealing the target's own position.

[0005] In a first aspect, embodiments of this application provide a target localization method, including:

[0006] A first image and a second image are acquired, which are respectively taken at a first position and a second position targeting the same target object; the distance between the first position and the second position is obtained through a ranging device; feature extraction and matching are performed on the first image and the second image respectively to obtain a first set of feature point pairs, each of which includes at least eight feature point pairs, each of which includes two feature points from the first image and the second image respectively, and the two feature points are matched; a first rotation matrix and a first translation vector are obtained based on the first set of feature point pairs; the orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position.

[0007] The two images were taken by two cameras located at different positions targeting the same object, or by one camera located at two different positions targeting the same object.

[0008] Optionally, the ranging device can be a laser rangefinder.

[0009] In this application, the distance between cameras is obtained through a ranging device, and the distance and orientation information of the target object are determined based on the acquired images and distances. Compared with existing technologies, this method does not expose the user's own position. Furthermore, this application calculates the distance of the target object based on two or more images, which reduces the computational load. This application obtains the relative positions between cameras at different locations through common field-of-view registration between images, overcoming the problem of low measurement accuracy due to short baselines in large outdoor scenes. It also eliminates the need for a calibration board, greatly reducing the complexity of the system. The ranging device in this application determines the distance and scale information between cameras, enabling subsequent calculation of the relative distance to the target object based on the baseline. This reduces errors caused by camera and laser rangefinder calibration and also lowers the hardware requirements for the positioning system; only two or more cameras are needed to complete target positioning. This application's solution can be applied to target positioning in large outdoor scenes, with low hardware requirements, high positioning accuracy, and wider applicability.

[0010] Secondly, embodiments of this application provide a positioning device, comprising:

[0011] The acquisition unit is used to acquire a first image and a second image, which are respectively captured at a first position and a second position targeting the same target object; and to acquire the distance between the first position and the second position obtained by a ranging device.

[0012] The feature processing unit is used to extract and match features from the first image and the second image respectively to obtain a first feature point pair set. Each feature point pair set includes at least eight feature point pairs. Each feature point pair includes two feature points from the first image and the second image respectively, and the two feature points are matched.

[0013] The determining unit is used to obtain a first rotation matrix and a first translation vector based on a first set of feature points; and to determine the orientation information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position.

[0014] Thirdly, embodiments of this application also provide a positioning device, including a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute part or all of the method described in the first aspect.

[0015] Fourthly, embodiments of this application also provide a chip system applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs part or all of the method described in the first aspect.

[0016] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that is executed by a processor to implement part or all of the method described in the first aspect.

[0017] Sixthly, embodiments of this application also provide a computer program that is executed to implement part or all of the methods described in the first aspect.

[0018] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1a A schematic diagram of a system architecture provided for an embodiment of this application;

[0021] Figure 1b Another system architecture diagram provided for an embodiment of this application;

[0022] Figure 2 A flowchart illustrating a target localization method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a pinhole imaging model;

[0024] Figure 4 This is a schematic diagram illustrating the principle of a measurement method based on line-of-sight intersection.

[0025] Figure 5 This is a schematic diagram of the structure of a positioning device provided in an embodiment of this application;

[0026] Figure 6This is a schematic diagram of another positioning device provided in an embodiment of this application. Detailed Implementation

[0027] The following sections will provide detailed explanations.

[0028] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0031] The embodiments of this application will now be described with reference to the accompanying drawings.

[0032] See Figure 1a , Figure 1a This is a schematic diagram of a system architecture provided for an embodiment of this application. Figure 1a As shown, the system includes a positioning device 101, a camera 102, and a ranging device 103. The camera 102 captures images of the same target object at a first position and a second position, respectively, to obtain a first image and a second image. The ranging device 103 acquires the distance between the first position and the ranging device 103, and the distance between the second position and the ranging device 103. The positioning device 101 acquires the distances between the first and second positions and the ranging device 103, respectively, and the two images. Based on these distances, it determines the distance between the first and second positions, respectively, to obtain a distance value. Finally, based on the two images and the distance value, it determines the location information of the target object.

[0033] Optionally, the camera 102 captures a third image of the target object at a third position; the ranging device 103 obtains the distance between the third position and the ranging device 103; the positioning device 101 further determines the distance between the first position and the third position, and the distance between the first position and the second position, based on the distances between the first position, the second position and the third position and the ranging device 103, to obtain two distance values; the positioning device determines the position information of the target object based on the three distance values ​​and the three images.

[0034] In another example, such as Figure 1b As shown, the system includes a positioning device 101, a ranging device 103, and two cameras 102, which are designated as a first camera and a second camera, respectively. The two cameras 102 are located at the same position. Both cameras 102 capture images of the same target object to obtain a first image and a second image. The ranging device 103 acquires the distances between the two cameras 102 and the ranging device 103. The positioning device 101 obtains the first image, the second image, and the distances between the two cameras 102 and the ranging device 103. The positioning device 101 determines the distance between the two cameras 102 based on the distances between them, thus obtaining a distance value. The system then determines the location information of the target object based on the first image, the second image, and the distance value.

[0035] Optionally, the system also includes a third camera, the location of which is different from that of the first and second cameras; the third camera captures an image of the target object to obtain a third image; the ranging device 103 obtains the distance between the third camera and the ranging device 103; the positioning device 101 determines the distance between the first and third cameras based on the distance between the third camera and the ranging device 103, and the distance between the first camera and the ranging device 103; the positioning device 101 determines the distance between the first and third cameras based on the distance between the third camera and the ranging device 103, and the distance between the second camera and the ranging device 103, obtaining two distance values; the positioning device 101 determines the position information of the target image based on the three distance values ​​and the three images.

[0036] See Figure 2 , Figure 2 This is a flowchart illustrating a target localization method provided in an embodiment of this application. Figure 2 As shown, the method includes:

[0037] S201, the positioning device acquires a first image and a second image, which are respectively taken at a first position and a second position targeting the same target object; and acquires the distance between the first position and the second position obtained by the ranging device.

[0038] Specifically, the first and second images can be obtained by a single camera capturing images of the target object from two different locations, or by two cameras capturing images of the target object from two different locations. A ranging device collects the distance between itself and a single camera at the two different locations, obtaining two distance values; the positioning device determines the distance between the two different locations based on these two distance values. Alternatively, the ranging device collects the distance between itself and two cameras located at two different locations, obtaining two distance values; the positioning device determines the distance between each pair of cameras based on these two distance values.

[0039] S202, The positioning device performs feature extraction and matching on the first image and the second image respectively to obtain a first set of feature point pairs.

[0040] The first feature point pair set includes at least eight feature point pairs, each pair consisting of two feature points from two of the three images, and the two feature points are matched.

[0041] In one feasible embodiment, feature extraction and matching are performed on the first image and the second image respectively to obtain a first set of feature point pairs, including:

[0042] Feature extraction is performed on the first image to obtain multiple first feature points; feature extraction is performed on the second image to obtain multiple second feature points;

[0043] Multiple first feature points and multiple second feature points are matched to obtain a set of first feature point pairs; each feature point pair in the set of first feature point pairs includes two matched feature points, which are respectively from the first image and the second image.

[0044] Specifically, the localization device extracts features from the first image and the second image respectively to obtain feature points for each image. The feature extraction methods include, but are not limited to, the Speeded-Up Robust Features (SURF) extraction method, the Scale-Invariant Feature Transform (SIFT) extraction method, the Features from Accelerated Segment Test (FAST) extraction method, and the Binary Robust Independent Elementary Features (BRIEF) extraction method. The localization device matches the feature points of the first image with the feature points of the second image to obtain a first set of feature point pairs. The first set of feature point pairs includes at least eight pairs, with each pair containing two feature points from the first image and the second image, respectively.

[0045] Optionally, the positioning device performs feature extraction on the third image to obtain the third feature points of the third image. After obtaining the feature points of each of the three images, the positioning device matches the feature points of each pair of the three images to obtain a set of three feature point pairs; for example, the three images are image A, image B, and image C; the positioning device matches the feature points of image A with the feature points of image B to obtain a set of feature point pairs of image A and image B. This set of feature point pairs includes at least eight feature point pairs, each of which includes two feature points from image A and image B respectively, and these two feature points are matched; similarly, the feature points of image B and image C are processed in the same way to obtain a set of feature point pairs of image B and image C; the feature points of image A and image C are processed to obtain a set of feature point pairs of image A and image C.

[0046] Optionally, two matching feature points are defined as two feature points whose Euclidean distance is less than a preset threshold.

[0047] Optionally, for images A and B, the distance between any feature point F in image A and every feature point in image B is determined. Based on the distance between feature point F in image A and every feature point in image B, at least one reference feature point is determined from image B, where the Euclidean distance between each of these at least one first reference feature point and feature point F is less than a preset threshold. The feature point with the smallest Euclidean distance between the at least one first reference feature point and feature point F is determined as the feature point that matches feature point F. If no first reference feature point is determined from image B based on the distance between feature point F in image A and every feature point in image B, it indicates that no feature point in image B matches feature point F. Following this method, a set of feature point pairs for images A and B can be determined. This method can improve the accuracy of feature point matching.

[0048] Optionally, feature points in image A are clustered to obtain at least one first cluster center; feature points in image B are clustered to obtain at least one second cluster center; the Euclidean distance between any first cluster center T and each of the at least one second cluster center is determined; at least one third cluster center is determined from the at least one second cluster center based on the Euclidean distance between the first cluster center T and each of the at least one second cluster center, wherein the Euclidean distance between each third cluster center and the first cluster center F is less than a preset threshold; then the Euclidean distance between any feature point G among the feature points corresponding to the first cluster center and each feature point among the feature points corresponding to the third cluster center is determined; at least one second reference feature point is determined from each feature point among the feature points corresponding to the third cluster center based on the Euclidean distance between feature point G and each feature point among the feature points corresponding to the third cluster center, wherein the Euclidean distance between each of the at least one second reference feature point and feature point G is less than a preset threshold; the second reference feature point with the smallest Euclidean distance to feature point G among the at least one second reference feature point is determined as the feature point matching feature point G. Following the method described above, the set of feature point pairs for image A and image B can be determined. This approach can improve both the accuracy and efficiency of feature point matching.

[0049] It should be noted that the feature points corresponding to the first cluster center refer to the feature points obtained by clustering to the first cluster center; the feature points corresponding to the third cluster center refer to the feature points obtained by clustering to the third cluster center.

[0050] S203, the positioning device obtains the first rotation matrix and the first translation vector based on the first set of feature points.

[0051] In a feasible embodiment, obtaining a first rotation matrix and a first translation vector based on a first set of feature point pairs includes:

[0052] The first essential matrix is ​​determined based on the first feature point; the first essential matrix is ​​decomposed into singular values ​​to obtain the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix; the first rotation matrix is ​​obtained based on the left singular matrix and the right singular matrix of the first essential matrix; the first translation vector is determined based on the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix.

[0053] The following section details how to obtain a rotation matrix and translation vector based on a set of feature point pairs.

[0054] Specifically, the coordinates of the feature points of the target object in the world coordinate system are [X... w Y w Z w ] T The coordinates of this feature point in the first and second images are P1 = [X...]. C1 Y C1 Z C1 ] T P2 = [X C2 Y C2 Z C2 ] T These two coordinates can also be viewed as the coordinates of the feature point in the coordinate systems of the camera capturing the first image and the camera capturing the second image. Based on the coordinates of the target object's feature point in the world coordinate system and the coordinates of that feature point in the two images, the relationship between the world coordinate system and the camera coordinate system can be determined as follows:

[0055]

[0056] R1 is the rotation matrix between the world coordinate system and the coordinate system of the camera that took the first image, R2 is the rotation matrix between the world coordinate system and the coordinate system of the camera that took the second image, t1 is the translation vector between the world coordinate system and the coordinate system of the camera that took the first image, and t2 is the translation vector between the world coordinate system and the coordinate system of the camera that took the second image.

[0057] Assume the transformation relationship between the coordinate systems of the cameras that took the above two images is as follows:

[0058] P2 = R 12 P1+t 12

[0059] Among them, R 12 Let t be the rotation matrix between the two images mentioned above. 12 Let be the translation vector between the two images mentioned above.

[0060] It should be noted that two feature points with coordinates P1 and P2 in the two images constitute a feature point pair, which corresponds to a feature point of the target object. In other words, there is one feature point in each image that corresponds to a feature point of the target object.

[0061] Since the coordinates of P1 and P2 are unknown, but their positions in the image are known, we can determine their positions based on the given information. Figure 3 The pinhole imaging model shown determines the correspondence between the pixel coordinate system and the camera coordinate system:

[0062]

[0063]

[0064] in, and The intrinsic parameter matrices of the cameras that captured the two images can be pre-calibrated using methods including, but not limited to, Zhang Zhengyou's calibration method. Therefore, based on the relationship between the pixel coordinate system and the camera coordinate system, and the transformation relationship between the coordinate systems of the cameras that captured the two images, the following formula is obtained:

[0065]

[0066] By normalizing the Z-axis of the camera coordinate system, the number of unknowns is reduced, making it easier to calculate the rotation matrix and translation vector between the cameras that captured the two images, i.e.:

[0067]

[0068]

[0069]

[0070] Therefore, the rotation matrix and translation vector between the two cameras, which lack scale information, can be calculated based on the epipolar geometry constraints:

[0071] P2'=R 12 P1'+t 12

[0072] t 12 ×P2'=t 12 ×R 12 P1'

[0073] P2' T (t 12 ×P2')=P2' T (t 12 ×R 12 P1')=0

[0074] Let E = t 12 ×R 12 Then we have:

[0075] P2' T EP1'=0

[0076] Since matrix E is an essential matrix, the above equation is equivalent to:

[0077]

[0078] Here, matrix F is the basic matrix. Since the camera's intrinsic parameters have been determined through camera pre-calibration, the pixel coordinate system can be transformed to the camera coordinate system by left-multiplying by the intrinsic parameters matrix K, that is:

[0079]

[0080]

[0081] The localization device determines the essential matrix based on the coordinates of a feature point in the set of feature points in two images, that is, by solving the following equation based on the coordinates of a feature point in the set of feature points:

[0082]

[0083] The above formula can be simplified to:

[0084]

[0085] Here, e1,…,e9 are the nine unknowns of the essential matrix E. The essential matrix has five degrees of freedom, but since these five degrees of freedom are non-linearly related, solving it using the coordinates of the feature points in five feature point pairs is quite complex. Therefore, we use the coordinates of the feature points in eight feature point pairs to solve it:

[0086]

[0087] in, and These are the normalized pixel coordinates of the two feature points in the first feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the second feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the third feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the fourth feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the 5th feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the 6th feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the 7th feature point pair in their respective images; and These are the normalized pixel coordinates of the two feature points in the 8th feature point pair within their respective images.

[0088] The positioning device solves the linear equation Ae = 0 to obtain the vector e, thereby determining the essential matrix E. Generally, by solving the essential matrix based on a large number of feature points, a least-squares solution can be found: the vector e is a square matrix A. T The smallest vector corresponding to the smallest eigenvalue of A.

[0089] After obtaining the essential matrix E using the least squares method, the rotation matrix R and translation vector t between the cameras in the two images are obtained by performing SVD decomposition on the essential matrix E.

[0090] The essential matrix E is decomposed by SVD to obtain: E = UΣV; U is the left singular matrix, Σ is the singular matrix, and V is the right singular matrix.

[0091] The positioning device can determine the rotation matrix R and translation vector t based on the following formula.

[0092]

[0093]

[0094] in, Let R be a matrix that rotates 90 degrees around the Z-axis (positive or negative). By verification, a unique solution can be determined, which yields the unique rotation matrix R and translation vector t.

[0095] S204. The positioning device determines the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position.

[0096] In one feasible embodiment, the positioning device determines the orientation information and distance of the target object based on a first translation vector, a first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, including:

[0097] First scale information is determined based on the first translation vector and the distance between the first and second positions; first reference coordinates of the target object are determined based on the first scale information, the first rotation matrix, and the coordinates of the target object in the first and second images respectively, the first reference coordinates being the coordinates of the target object in the camera coordinate system; the first reference coordinates of the target object are processed based on the first scale information to obtain the first true coordinates of the target object; the distance of the target object is determined based on the first true coordinates; the direction information of the target object is determined based on the northward information and the positioning information of the ranging device, the direction information of the target object being used to indicate the direction of the target object relative to the ranging device.

[0098] For images A and B, binocular correction is performed on the cameras that captured image A and image B based on the rotation matrix between them, so that the imaging planes of the cameras capturing image A and image B are coplanar, and the epipolar lines are aligned. After binocular correction, the coordinates of the target object are determined according to the principle of triangulation.

[0099] like Figure 4 As shown, assume that the image points of the target object in the image planes of the camera that captured image A and the camera that captured image B are Pl and Pr, respectively; f is the focal length of the camera; D is the distance between the camera's position when image A is captured and the camera's position when image B is captured; Z is the distance between the target object and the baseline; according to the triangle principle, the coordinates (X, YZ) of the target object in the camera coordinate system can be determined, and these coordinates are the reference coordinates.

[0100]

[0101]

[0102]

[0103]

[0104] Among them, u A and v A These are the x and y coordinates of the target object in image A, respectively; u B and v B Let x and y be the x and y coordinates of the target object in image B, respectively, and let Cl and Cr be the optical origins of the target object in the camera that captured image A and the camera that captured image B, respectively.

[0105] The positioning device obtains the target object's coordinates (X, Y, Z) in the camera coordinate system as described above. Since these coordinates lack scale information, scale information needs to be introduced. Based on this scale information and the target object's coordinates (X, Y, Z) in the camera coordinate system, the true coordinates (X, Y, Z) of the target object are determined. AB ,Y AB Z AB The scale information is determined based on the translation vector between image A and image B, and the distance between the camera's position when image A is taken of the target object and the camera's position when image B is taken of the target object.

[0106]

[0107]

[0108]

[0109] After obtaining the true coordinates of the target object, the positioning device can determine the distance L between the target object and the ranging device. AB .in, The relative direction of the target object with respect to the ranging device is determined based on northward information and positioning information.

[0110] In one feasible embodiment, the method of this application further includes:

[0111] A third image is acquired, which is taken of the target object at a third position. The distance between the third position and the second position, and the distance between the third position and the first position are obtained using a ranging device. Features are extracted from the third image to obtain multiple third feature points. The extracted third feature points are then matched with feature points from the first image and the second image to obtain a set of second feature point pairs and a set of third feature point pairs. A second rotation matrix and a second translation vector are obtained based on the set of second feature point pairs. A third rotation matrix and a third translation vector are obtained based on the set of third feature point pairs.

[0112] The orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, including:

[0113] The orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the second translation vector, the second rotation matrix, the third translation vector, the third rotation matrix, the coordinates of the target object in the first image, the second image, and the third image, and the distances between each pair of positions in the first position, the second position, and the third position.

[0114] Optionally, the positioning device acquires a third image, which is a picture of the target object taken by one of the cameras at a third position; or a picture of the target object taken by a third camera at a third position different from the two aforementioned positions.

[0115] Optionally, the positioning device performs feature extraction on the third image to obtain the third feature points of the third image. After obtaining the feature points of each of the three images, the positioning device matches the feature points of each pair of the three images to obtain a set of three feature point pairs; for example, the three images are image A, image B, and image C; the positioning device matches the feature points of image A with the feature points of image B to obtain a set of feature point pairs of image A and image B. This set of feature point pairs includes at least eight feature point pairs, each of which includes two feature points from image A and image B respectively, and these two feature points are matched; similarly, the feature points of image B and image C are processed in the same way to obtain a set of feature point pairs of image B and image C; the feature points of image A and image C are processed to obtain a set of feature point pairs of image A and image C.

[0116] Optionally, based on three sets of feature point pairs, three rotation matrices and three translation vectors can be obtained corresponding to the three feature point pairs; that is, for a set of feature point pairs, one rotation matrix and one translation vector can be obtained. For example, the rotation matrix and translation vector obtained based on the feature point pairs of image A and image B are respectively the rotation matrix and translation vector between image A and image B; the rotation matrix and translation vector obtained based on the feature point pairs of image B and image C are respectively the rotation matrix and translation vector between image B and image C.

[0117] Similarly, following the above method, the distance L between the target object and the ranging device can be determined based on the rotation matrix and translation vector between image A and image C, the coordinates of the target object in image A and image C, and the distance between the camera's position when image A is taken and the camera's position when image C is taken. AC The distance L between the target object and the ranging device can be determined based on the rotation matrix and translation vector between images B and C, the coordinates of the target object in image B and image C, and the distance between the camera's position when image B was taken and the camera's position when image C was taken. BC For L AB L AC and L BC Processing methods, such as averaging or weighted averaging, are used to obtain the distance to the target object.

[0118] In a feasible embodiment, the true coordinates (X) of the target object are obtained in the manner described above. AB ,Y AB Z AB ), (X AC ,Y AC Z AC ) and (X BC ,Y BC Z BC ); for X AB X AC and X BC Processing is performed, including but not limited to averaging or weighted averaging, to obtain X. 目标 ; For Y AB Y AC and Y BC Processing is performed, including but not limited to averaging or weighted averaging, to obtain Y. 目标 ; For Z AB Z AC and Z BC Processing is performed, including but not limited to averaging or weighted averaging, to obtain Z. 目标 According to X 目标 Y 目标 and Z 目标 Obtain the distance L of the target object 目标 .

[0119] in,

[0120] It should be noted that image A is the first image, image B is the second image, and image C is the third image.

[0121] It should be noted that the solution of this application can also be based on four or more images for localization; for specific implementation, please refer to the process of localization based on three images. Specifically, for four or more images, it is not necessary to pair feature points from each of the four or more images. For example, with four images, it is only necessary to pair feature points from the first and second images, the second and third images, and the third and fourth images to obtain three sets of feature point pairs. Optionally, feature points from the first and fourth images can also be paired to obtain one set of feature point pairs. The distance used during localization corresponds to the set of feature point pairs. For example, if the set of feature point pairs obtained by pairing feature points from the first and second images is used, then the distance used for localization is the distance between the position where the camera captured the first image and the position where the camera captured the second image.

[0122] By introducing more images, the accuracy of positioning can be further improved.

[0123] As can be seen, most existing technical solutions use laser rangefinders to detect targets. Laser rangefinders can directly obtain distance information between the target and the laser. However, on the battlefield, the enemy will also use laser detectors to detect the laser for counter-reconnaissance. In the solution of this application, the target's location information is obtained indirectly through a laser rangefinder, which effectively avoids being detected by the enemy's laser detector when directly detecting the target with a laser rangefinder, thus exposing the scout's position. For binocular or multi-view vision measurement systems, fixed modules are commonly used to fix the cameras, and specific calibration plates are designed to calibrate the pose relationship between the cameras. Due to hardware limitations, these systems have short baselines and offer good accuracy for close-up or indoor measurements, but significant errors occur when measuring large outdoor scenes at long distances. This application utilizes rich outdoor texture information for camera registration. Specifically, it uses matching feature points between images to determine the rotation matrix between them, and then uses the rotation matrix to align the imaging planes of the cameras capturing the two images. This reduces the pre-registration steps and is not limited by hardware, allowing for a more reasonable determination of the baseline between cameras. The ranging device in this application determines the distance and scale information between the cameras, enabling subsequent calculations of the relative distance to the target object based on the baseline. This reduces errors caused by camera and laser rangefinder calibration and also lowers the hardware requirements for the positioning system, requiring only two or more cameras to complete target localization. Furthermore, by using two or more images to calculate the position and orientation information of the target object, the computational load is reduced. This application obtains the relative positions between cameras at different locations through common field-of-view registration between images. This overcomes the problem of low measurement accuracy due to short baselines in large outdoor scenes, and also eliminates the need for a corresponding calibration board, greatly reducing the complexity of the system. The solution presented in this application can be applied to target localization in large outdoor scenes, with low hardware requirements, high positioning accuracy, and wider applicability.

[0124] See Figure 5 , Figure 5 This is a schematic diagram of a positioning device provided in an embodiment of this application. Figure 5 As shown, the positioning device 500 includes:

[0125] The acquisition unit 501 is used to acquire a first image and a second image, which are respectively captured at a first position and a second position targeting the same target object; and to acquire the distance between the first position and the second position obtained by a ranging device.

[0126] The feature processing unit 502 is used to extract and match features from the first image and the second image respectively to obtain a first feature point pair set. Each feature point pair set includes at least eight feature point pairs. Each feature point pair includes two feature points from the first image and the second image respectively, and the two feature points are matched.

[0127] The determining unit 503 is used to obtain a first rotation matrix and a first translation vector based on a first set of feature points; and to determine the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position.

[0128] In a feasible embodiment, in determining the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, the determining unit 503 is specifically used for:

[0129] First scale information is determined based on the first translation vector and the distance between the first and second positions; first reference coordinates of the target object are determined based on the first scale information, the first rotation matrix, and the coordinates of the target object in the first and second images respectively, the first reference coordinates being the coordinates of the target object in the camera coordinate system; the first reference coordinates of the target object are processed based on the first scale information to obtain the first true coordinates of the target object; the distance of the target object is determined based on the first true coordinates; the direction information of the target object is determined based on the northward information and the positioning information of the ranging device, the direction information of the target object being used to indicate the direction of the target object relative to the ranging device.

[0130] In a feasible embodiment, in terms of obtaining the first rotation matrix and the first translation vector based on the first set of feature points, the determining unit 503 is specifically used for:

[0131] The first essential matrix is ​​determined based on the first feature point set; the first essential matrix is ​​decomposed into singular values ​​to obtain the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix; the first rotation matrix is ​​obtained based on the left singular matrix and the right singular matrix of the first essential matrix; the first translation vector is determined based on the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix.

[0132] In one feasible embodiment, the feature processing unit 502 is specifically used for:

[0133] Feature extraction is performed on the first image to obtain multiple first feature points; feature extraction is performed on the second image to obtain multiple second feature points;

[0134] Multiple first feature points and multiple second feature points are matched to obtain a set of first feature point pairs; each feature point pair in the set of first feature point pairs includes two matched feature points, which are respectively from the first image and the second image.

[0135] In a feasible embodiment, the acquisition unit 501 is further configured to acquire a third image, which is obtained by taking a picture of the target object at a third position, and to acquire the distance between the third position and the second position and the distance between the third position and the first position by a ranging device.

[0136] The feature processing unit 502 is also used to extract features from the third image to obtain multiple third feature points; and to match the extracted third feature points with feature points from the first image and feature points from the second image to obtain a set of second feature point pairs and a set of third feature point pairs.

[0137] The determining unit 503 is also used to obtain a second rotation matrix and a second translation vector based on the second feature point pair set; and to obtain a third rotation matrix and a third translation vector based on the third feature point pair set.

[0138] In determining the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, the determining unit 503 is specifically used for:

[0139] The orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the second translation vector, the second rotation matrix, the third translation vector, the third rotation matrix, the coordinates of the target object in the first image, the second image, and the third image, and the distances between each pair of positions in the first position, the second position, and the third position.

[0140] It should be noted that the above-mentioned units (acquisition unit 501, feature processing unit 502, and determination unit 503) are used to execute the relevant steps of the above method. For example, acquisition unit 501 is used to execute the relevant content of S201, feature processing unit 502 is used to execute the relevant content of S202, and determination unit 503 is used to execute the relevant content of S203 and S204.

[0141] In this embodiment, the positioning device 500 is presented in the form of a unit. Here, "unit" can refer to an application-specific integrated circuit (ASIC), a processor and memory executing one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above-mentioned functions. Furthermore, the acquisition unit 501, feature processing unit 502, and determination unit 503 described above can be... Figure 6 The positioning device shown is implemented by a processor 601.

[0142] like Figure 6 The positioning device 600 shown can be used to Figure 6 The positioning device 600 is implemented using the structure described above. It includes at least one processor 601, at least one memory 602, and at least one communication interface 603. The processor 601, the memory 602, and the communication interface 603 are connected through the communication bus and communicate with each other.

[0143] Processor 601 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs in the above scheme.

[0144] Communication interface 603 is used to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Networks (WLAN), etc.

[0145] Memory 602 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processor via a bus. Memory may also be integrated with the processor.

[0146] The memory 602 stores the application code for executing the above scheme, and its execution is controlled by the processor 601. The processor 601 executes the application code stored in the memory 602.

[0147] The code stored in memory 602 can execute any of the target localization methods provided above, such as:

[0148] A first image and a second image are acquired, which are respectively taken at a first position and a second position targeting the same target object; the distance between the first position and the second position is obtained through a ranging device; feature extraction and matching are performed on the first image and the second image respectively to obtain a first set of feature point pairs, each of which includes at least eight feature point pairs, each of which includes two feature points from the first image and the second image respectively, and the two feature points are matched; a first rotation matrix and a first translation vector are obtained based on the first set of feature point pairs; the orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position.

[0149] This application also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, includes some or all of the steps of any of the target positioning methods described in the above method embodiments.

[0150] This application also provides a computer program that is executed to implement some or all of the steps of any of the target localization methods described in the above method embodiments.

[0151] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0152] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0158] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal target localization method, characterized in that, include: Acquire a first image and a second image, wherein the first image and the second image are respectively captured at a first position and a second position targeting the same object; And obtain the distance between the first and second positions obtained through the ranging device; Feature extraction and matching are performed on the first image and the second image respectively to obtain a first feature point pair set. Each of the first feature point pair set includes at least eight feature point pairs. The feature point pairs include two feature points from the first image and the second image respectively, and the two feature points are matched. The first rotation matrix and the first translation vector are obtained based on the first set of feature points; A third image is acquired, which is taken of the target object at a third position, and the distance between the third position and the second position, and the distance between the third position and the first position are obtained by the ranging device. Feature extraction is performed on the third image to obtain multiple third feature points; and the extracted third feature points are matched with feature points of the first image and feature points of the second image respectively to obtain a set of second feature point pairs and a set of third feature point pairs. The second rotation matrix and the second translation vector are obtained based on the second set of feature points; the third rotation matrix and the third translation vector are obtained based on the third set of feature points. The orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position. Specifically, the orientation information and distance of the target object are determined based on the first translation vector, the first rotation matrix, the second translation vector, the second rotation matrix, the third translation vector, the third rotation matrix, the coordinates of the target object in the first image, the second image and the third image respectively, and the distance between each pair of positions in the first position, the second position and the third position.

2. The method according to claim 1, characterized in that, The step of determining the orientation information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position includes: The first scale information is determined based on the first translation vector and the distance between the first position and the second position. Based on the first scale information, the first rotation matrix, and the coordinates of the target object in the first image and the second image respectively, the first reference coordinates of the target object are determined, and the first reference coordinates are the coordinates of the target object in the camera coordinate system. The first reference coordinates of the target object are processed according to the first scale information to obtain the first true coordinates of the target object; The distance to the target object is determined based on the first real coordinates; The direction information of the target object is determined based on the northward information and the positioning information of the ranging device. The direction information of the target object is used to indicate the direction of the target object relative to the ranging device.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the first rotation matrix and the first translation vector based on the first set of feature points includes: The first essential matrix is ​​determined based on the set of first feature points; Singular value decomposition is performed on the first essential matrix to obtain the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix; The first rotation matrix is ​​obtained based on the left and right singular matrices of the first essential matrix; the first translation vector is determined based on the left, singular, and right singular matrices of the first essential matrix.

4. The method according to claim 1 or 2, characterized in that, The step of extracting and matching features from the first image and the second image respectively to obtain a first set of feature point pairs includes: Feature extraction is performed on the first image to obtain multiple first feature points; feature extraction is performed on the second image to obtain multiple second feature points; The plurality of first feature points and the plurality of second feature points are matched to obtain a first feature point pair set; each feature point pair in the first feature point pair set includes two matched feature points, the two matched feature points being from the first image and the second image, respectively.

5. A positioning device, characterized in that, include: The acquisition unit is used to acquire a first image and a second image, wherein the first image and the second image are respectively captured at a first position and a second position targeting the same target object; And obtain the distance between the first and second positions obtained through the ranging device; A feature processing unit is configured to perform feature extraction and matching on the first image and the second image respectively to obtain a first feature point pair set. Each of the first feature point pair set includes at least eight feature point pairs. The feature point pairs include two feature points from the first image and the second image respectively, and the two feature points are matched. The acquisition unit is further configured to acquire a third image, which is obtained by taking a picture of the target object at a third position, and to acquire the distance between the third position and the second position, and the distance between the third position and the first position through the ranging device; The feature processing unit is further configured to extract features from the third image to obtain multiple third feature points; and to match the extracted third feature points with the feature points of the first image and the feature points of the second image respectively to obtain a set of second feature point pairs and a set of third feature point pairs. The determining unit is further configured to obtain a second rotation matrix and a second translation vector based on the second feature point pair set; and to obtain a third rotation matrix and a third translation vector based on the third feature point pair set. The determining unit is configured to obtain a first rotation matrix and a first translation vector based on the first feature point pair set; and to determine the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, specifically including: determining the direction information and distance of the target object based on the first translation vector, the first rotation matrix, the second translation vector, the second rotation matrix, the third translation vector, the third rotation matrix, the coordinates of the target object in the first image, the second image and the third image respectively, and the distance between each pair of positions in the first position, the second position and the third position.

6. The apparatus according to claim 5, characterized in that, In determining the orientation information and distance of the target object based on the first translation vector, the first rotation matrix, the coordinates of the target object in the first image and the second image respectively, and the distance between the first position and the second position, the determining unit is specifically used for: The first scale information is determined based on the first translation vector and the distance between the first position and the second position. Based on the first scale information, the first rotation matrix, and the coordinates of the target object in the first image and the second image respectively, the first reference coordinates of the target object are determined, and the first reference coordinates are the coordinates of the target object in the camera coordinate system. The first reference coordinates of the target object are processed according to the first scale information to obtain the first true coordinates of the target object; The distance to the target object is determined based on the first real coordinates; The direction information of the target object is determined based on the northward information and the positioning information of the ranging device. The direction information of the target object is used to indicate the direction of the target object relative to the ranging device.

7. The apparatus according to claim 5 or 6, characterized in that, In the aspect of obtaining the first rotation matrix and the first translation vector based on the first set of feature points, the determining unit is specifically used for: The first essential matrix is ​​determined based on the set of first feature points; Singular value decomposition is performed on the first essential matrix to obtain the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix; The first rotation matrix is ​​obtained based on the left and right singular matrices of the first essential matrix; The first translation vector is determined based on the left singular matrix, the singular matrix, and the right singular matrix of the first essential matrix.

8. The apparatus according to any one of claims 5-6, characterized in that, The feature processing unit is specifically used for: Feature extraction is performed on the first image to obtain multiple first feature points; feature extraction is performed on the second image to obtain multiple second feature points; The plurality of first feature points and the plurality of second feature points are matched to obtain a first feature point pair set; each feature point pair in the first feature point pair set includes two matched feature points, the two matched feature points being from the first image and the second image, respectively.

Images