Multi-entity relative position determination method and device, equipment and medium

Abstract

The invention provides a multi-entity relative position determination method and device, equipment and a medium, and the method comprises the steps: obtaining image data of an environment where a corresponding entity is located through a visual sensor equipped for each entity in a plurality of entities; for any entity, performing feature point extraction on the image data corresponding to the entity to obtain a plurality of feature points corresponding to the entity; constructing a sparse map of the entity in the corresponding entity coordinate system based on the plurality of feature points corresponding to the entity; and aligning the sparse maps corresponding to any two entities in the plurality of entities to determine the relative position between any two entities. Therefore, by constructing the sparse map of each entity in the multiple entities and aligning the sparse maps, the relative positions among the multiple entities can be automatically determined, dependence on an external calibration object is not needed, adaptability is high, flexibility is high, adaptation cost and time are reduced, and robustness, accuracy and reliability of relative position determination are improved.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a method, device, equipment and medium for determining the relative positions of multiple entities. Background Art

[0002] In scenarios of multi-entity collaboration, whether it is collaborative robots, convoy driving of unmanned vehicles, or interaction of multiple terminal devices in a complex environment, determining the relative positions of these entities is crucial to the efficiency and security of multi-entity collaboration. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of the present application is to propose a method for determining the relative positions of multiple entities.

[0005] The second objective of the present application is to provide a device for determining relative positions of multiple entities.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned purpose, the first aspect of the present application proposes a method for determining the relative positions of multiple entities, including: acquiring image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity among multiple entities; for any entity, extracting feature points of the image data corresponding to the entity to obtain multiple feature points corresponding to the entity; based on the multiple feature points corresponding to the entity, constructing a sparse map of the entity in the corresponding entity coordinate system; aligning the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between any two entities.

[0010] To achieve the above-mentioned purpose, the second aspect of the present application proposes a device for determining the relative positions of multiple entities, including: an acquisition module, used to acquire image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity among multiple entities; an extraction module, used to extract feature points from the image data corresponding to any of the entities, and obtain multiple feature points corresponding to the entity; a construction module, used to construct a sparse map of the entity based on the multiple feature points corresponding to the entity; and an alignment module, used to align the sparse maps corresponding to any two of the multiple entities to determine the relative position between any two entities.

[0011] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect embodiment above.

[0012] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment above.

[0013] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect of the present application.

[0014] The method, apparatus, device and medium for determining the relative positions of multiple entities provided in the present application can automatically determine the relative positions of multiple entities by constructing a sparse map of each entity in the multiple entities and aligning the sparse maps. It does not rely on external calibration objects and is suitable for various complex and dynamic environments. It has strong adaptability and high flexibility. Since it does not require pre-definition of patterns or features of calibration objects, it reduces the complexity of algorithm development and adaptation, thereby reducing adaptation costs and time. Even when some feature points are blocked or the environment changes, the sparse map can still be aligned and positioned through the remaining feature points, thereby improving the robustness, accuracy and reliability of relative position determination.

[0015] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application; Figure 3 A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application; Figure 4 A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application; Figure 5A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application; Figure 6 A schematic diagram of the structure of a device for determining relative positions of multiple entities provided in another embodiment of the present application. DETAILED DESCRIPTION

[0017] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0018] In the related art, in the process of determining the relative positions between multiple entities, external markers (such as a QR code with prior information) or calibration plates are usually relied on, that is, multiple entities with visual sensors observe the same object with features to achieve the determination of the relative positions between multiple entities. Specifically, the following steps may be included: 1. External calibration object: Place an object with known features in the working environment. The object can be a QR code or a calibration plate, etc. These objects usually have feature points that are easy to identify and calculate, such as the positioning mark of the QR code or the specific pattern on the calibration plate.

[0019] 2. Relative position calculation: The relative position between each entity and the calibration object can be calculated, and the calibration algorithm can be used to determine the positional relationship between the calibration object and the corresponding entity; further, after obtaining the positional relationship between each entity and the calibration object, the position of the calibration object can be used as an intermediary to indirectly deduce the relative positions between multiple entities.

[0020] However, this method has the following disadvantages: 1. Dependence on external markers: Calibration objects need to be placed in the environment, which limits the flexibility of usage scenarios.

[0021] 2. The marking type needs to be defined in advance: Taking the calibration plate as an example, the pattern drawn on the calibration plate needs to be accurately defined in advance, and the algorithm needs to be adapted according to this definition, which has a high cost for algorithm adaptation.

[0022] 3. Poor reliability: When the environment changes or the calibration object is blocked, the reliability of the system will be affected.

[0023] In response to at least one of the above problems, the present application proposes a method, device, equipment and medium for determining relative positions of multiple entities.

[0024] The following describes the method, device, equipment and medium for determining the relative positions of multiple entities in embodiments of the present application with reference to the accompanying drawings.

[0025] Figure 1 A schematic diagram of a flow chart of a method for determining relative positions of multiple entities provided in an embodiment of the present application.

[0026] The embodiment of the present application takes the method for determining relative positions of multiple entities as an example in which the method is configured in a device for determining relative positions of multiple entities. The device for determining relative positions of multiple entities can be applied to any electronic device so that the electronic device can perform a function for determining relative positions of multiple entities.

[0027] Among them, the electronic device can be any device with computing capabilities, such as a personal computer (PC), an industrial computer, a host computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0028] like Figure 1 As shown, the method comprises the following steps: Step S101 , obtaining image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity among multiple entities.

[0029] It should be noted that this application does not place too many restrictions on entities. For example, in a robot collaborative operation scenario, the entity may be a robot, in a driverless vehicle platoon driving scenario, the entity may be a vehicle, in a multiple terminal device interaction scenario, the entity may be a terminal device, and so on.

[0030] Among them, the visual sensor can be a monocular camera, a binocular camera, an RGB-D (Red-Green-Blue-Depth) camera, etc., and this application does not impose any restrictions on this.

[0031] In an embodiment of the present application, each of the multiple entities is equipped with a corresponding visual sensor, so that image data of the environment in which the corresponding entity is located can be collected and acquired through the visual sensor equipped by each of the multiple entities.

[0032] As a possible implementation manner, the image data may include a plurality of continuous image frames.

[0033] Step S102: for any entity, extract feature points from the image data corresponding to the entity to obtain a plurality of feature points corresponding to the entity.

[0034] Among them, feature points may include corner points, edge points, spots, etc., and this application does not impose any restrictions on this.

[0035] In the embodiment of the present application, for any entity, feature points can be extracted from image data corresponding to the entity to obtain multiple feature points corresponding to the entity.

[0036] Optionally, a feature point detection algorithm may be used to detect or identify feature points in the image data, and the detected or identified feature points may be described to obtain corresponding feature point descriptors. For example, a SIFT (Scale-Invariant Feature Transform) algorithm, SURF (Speeded-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF‌‌), etc. may be used to detect feature points in the image data, and the detected feature points may be described to obtain corresponding feature point descriptors.

[0037] Optionally, before extracting feature points from the image data corresponding to the entity, the image data may be preprocessed, wherein the preprocessing may include denoising, grayscale conversion, etc., which is not limited in this application. This facilitates subsequent processing of the image data and improves the accuracy and effectiveness of feature point extraction.

[0038] Step S103: construct a sparse map of the entity based on multiple feature points corresponding to the entity.

[0039] In an embodiment of the present application, a plurality of feature points corresponding to an entity may be used to construct a sparse map of the entity.

[0040] It should be noted that each entity has a corresponding entity coordinate system, wherein the entity coordinate system may be, for example, a world coordinate system of the corresponding entity.

[0041] As an example, the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system can be obtained, so that the feature points corresponding to the entity can be fused based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system to obtain a sparse map of the entity.

[0042] Step S104: aligning the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between any two entities.

[0043] The relative position may include six-degree-of-freedom data, that is, the position and rotation angle in space, and the relative distance, etc. For example, in a three-dimensional space, the coordinate system of the three-dimensional space is O-XYZ, and the relative position may include the position x in the X-axis direction, the position y in the Y-axis direction, the position z in the Z-axis direction, and the rotation angle Roll around the Z axis, the rotation angle Pitch around the Y axis, the rotation angle Yaw around the X axis, and the relative distance d.

[0044] The method for determining the relative position of multiple entities in the embodiment of the present application obtains image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity in the multiple entities; for any entity, extracts feature points from the image data corresponding to the entity to obtain multiple feature points corresponding to the entity; based on the multiple feature points corresponding to the entity, constructs a sparse map of the entity; aligns the sparse maps corresponding to any two entities in the multiple entities to determine the relative position between any two entities. Thus, by constructing a sparse map of each entity in the multiple entities and aligning the sparse maps, the relative position between multiple entities can be automatically determined without relying on external calibration objects, and is suitable for various complex and dynamic environments, with strong adaptability and high flexibility. Since there is no need to predefine the pattern or features of the calibration object, the complexity of algorithm development and adaptation is reduced, thereby reducing the adaptation cost and time; even when some feature points are blocked or the environment changes, the sparse map can still be aligned and positioned through the remaining feature points, thereby improving the robustness, accuracy and reliability of relative position determination.

[0045] In the case where any two entities include a first entity and a second entity, in order to clearly illustrate in the above embodiment of the present application how the sparse maps corresponding to any two entities among multiple entities are aligned to determine the relative positions between any two entities, the present application also proposes a method for determining the relative positions of multiple entities.

[0046] Figure 2 A schematic flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application.

[0047] like Figure 2 As shown, the method comprises the following steps: Step S201 , obtaining image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity among multiple entities.

[0048] Step S202: for any entity, extract feature points from the image data corresponding to the entity to obtain a plurality of feature points corresponding to the entity.

[0049] Step S203: construct a sparse map of the entity based on multiple feature points corresponding to the entity.

[0050] It should be noted that the execution process of step S201 to step S203 can refer to the relevant description in any embodiment of the present application and will not be repeated here.

[0051] Step S204: obtaining the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

[0052] As a possible implementation, Figure 3As shown, the following steps may be used to determine the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities: Step S2041, determining a first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity.

[0053] In an embodiment of the present application, for any feature point in the sparse map of the first entity, the feature point can be matched with each feature point in the sparse map of the second entity. When there is a first feature point among the feature points in the sparse map of the second entity that matches the feature point, it is determined that the feature point matches the feature point in the sparse map of the second entity. Thus, the feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity are accumulated to obtain a first quantity.

[0054] Among them, when matching any feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity, matching algorithms such as FLANN (Fast Library for ApproximateNearest Neighbors) and BFMatcher (Brute-Force Matcher) can be used.

[0055] For any feature point in the sparse map of the first entity, in order to accurately match the feature point with the feature points in the sparse map of the second entity, in a possible implementation method of the embodiment of the present application, for any feature point in the sparse map of the first entity, the distance between the feature point descriptor of the feature point and the feature point descriptor of any feature point in the sparse map of the second entity can be determined; when there is a second feature point in the sparse map of the second entity, and the distance between the feature point descriptor of the second feature point and the feature point descriptor of the feature point is less than a second set threshold, the second feature point is determined to be the first feature point that matches the feature point.

[0056] It should be noted that, for any feature point in the sparse map of the first entity, when determining the distance between the feature point descriptor of the feature point and the feature point descriptor of any feature point in the sparse map of the second entity, for example, a distance measurement method (such as Euclidean distance, Manhattan distance, cosine similarity, etc.) can be used for determination.

[0057] Among them, the second set threshold can be pre-set, and this application does not limit the value of the second set threshold.

[0058] Step S2042: determining a first coefficient according to the first quantity and the total quantity of feature points in the sparse map of the first entity.

[0059] As an example, a ratio of the first number to the total number of feature points in the sparse map of the first entity may be determined as the first coefficient.

[0060] For example, assuming that the total number of feature points in the sparse map of the first entity is a1, and the first number of feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity is b1, the first coefficient is (a1 / b1).

[0061] Step S2043, determining a second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity.

[0062] It should be noted that the method for determining the second number is similar to the method for determining the first number described above, and will not be elaborated here.

[0063] It should also be noted that the first quantity may be the same as the second quantity, or may be different, and this application does not impose any restrictions on this.

[0064] Step S2044: determine a second coefficient according to the second number and the total number of feature points in the sparse map of the second entity.

[0065] As an example, a ratio of the second number to the total number of feature points in the sparse map of the second entity may be determined as the second coefficient.

[0066] For example, assuming that the total number of feature points in the sparse map of the second entity is a2, and the second number of feature points in the sparse map of the second entity that match the feature points in the sparse map of the first entity is b2, the second coefficient is (a2 / b2).

[0067] It should be noted that the present application does not restrict the execution sequence of step S2041 to step S2042 and step S2043 to step S2044. The present application only takes step S2041 to step S2042 as an example of being executed before step S2043 to step S2044. Step S2041 to step S2042 can also be executed in parallel with step S2043 to step S2044, or, step S2041 to step S2042 can also be executed after step S2043 to step S2044.

[0068] Step S2045: determining the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities based on the first coefficient and / or the second coefficient.

[0069] As an example, the first coefficient may be determined as the similarity between a sparse map of a first entity and a sparse map of a second entity in any two entities.

[0070] As another example, the second coefficient may be determined as the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

[0071] As another example, the first coefficient and the second coefficient may be weighted averaged to obtain the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities.

[0072] Thus, the similarity between the sparse maps of any two entities can be determined efficiently and accurately.

[0073] Step S205 , in response to the similarity being greater than a first set threshold, aligning the sparse map of the first entity and the sparse map of the second entity to obtain a first relative posture between the first entity and the second entity.

[0074] Among them, the first set threshold can be pre-set, and the present application does not limit the value of the first set threshold.

[0075] The first relative posture may include a translation vector and a rotation matrix.

[0076] In an embodiment of the present application, when the similarity is greater than a first set threshold, it indicates that the first entity and the second entity of any two entities have observed similar scenes and a loop can be formed. At this time, the sparse map of the first entity and the sparse map of the second entity can be aligned to obtain a first relative posture between the first entity and the second entity.

[0077] As a possible implementation method, each feature point in the sparse map of the first entity and each feature point in the sparse map of the second entity can be matched to obtain a matched first feature point pair; based on the matched first feature point pair, the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is determined; based on the transformation relationship, the first relative posture between the first entity and the second entity is determined.

[0078] It should be noted that the present application does not limit the number of first feature point pairs.

[0079] In an embodiment of the present application, each feature point in the sparse map of the first entity can be matched with each feature point in the sparse map of the second entity to obtain a matched first feature point pair. For example, a feature matching algorithm (such as FLANN, BFMatcher and other matching algorithms) can be used to match each feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity to obtain a matched first feature point pair.

[0080] In an embodiment of the present application, the conversion relationship between the physical coordinate system of the first entity and the physical coordinate system of the second entity can be determined based on the matched first feature point pair. As an example, the spatial coordinates of each feature point in the matched first feature point pair in the corresponding physical coordinate system can be obtained, and then, one of the least squares method, iterative closest point (ICP) algorithm, etc. is used to determine the conversion relationship between the physical coordinate system of the first entity and the physical coordinate system of the second entity.

[0081] In the embodiment of the present application, the first relative posture between the first entity and the second entity can be determined based on the conversion relationship.

[0082] As an example, assume that the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is as shown in the following formula: ; (1) Among them, P1 represents the spatial coordinates of the feature point of the first entity in the physical coordinate system of the first entity, which is a vector with 3 rows and 1 column; P2 is the spatial coordinates of the feature point of the second entity in the physical coordinate system of the second entity, which is a vector with 3 rows and 1 column; R is a rotation matrix with 3 rows and 3 columns, which represents the rotation operation from the physical coordinate system of the second entity to the physical coordinate system of the first entity; T is a translation vector with 3 rows and 1 column, which represents the translation operation from the physical coordinate system of the second entity to the physical coordinate system of the first entity; Furthermore, according to formula (1), the first relative posture between the first entity and the second entity can be determined as follows: the rotation matrix of the first entity relative to the second entity is R, and the translation vector is T.

[0083] As another example, assume that the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity is as shown in the following formula: ; (2) Among them, P1 represents the spatial coordinates of the feature point of the first entity in the physical coordinate system of the first entity, which is a vector with 3 rows and 1 column; P2 is the spatial coordinates of the feature point of the second entity in the physical coordinate system of the second entity, which is a vector with 3 rows and 1 column; R' is a rotation matrix with 3 rows and 3 columns, which represents the rotation operation from the physical coordinate system of the second entity to the physical coordinate system of the first entity; T' is a translation vector with 3 rows and 1 column, which represents the translation operation from the physical coordinate system of the second entity to the physical coordinate system of the first entity; Furthermore, according to formula (2), the first relative posture between the first entity and the second entity can be determined as follows: the rotation matrix of the second entity relative to the first entity is R', and the translation vector is T'.

[0084] Therefore, by finding matching feature point pairs in the sparse map of any two entities, the transformation relationship between the entity coordinate systems of any two entities can be effectively determined, and then, based on the transformation relationship, the first relative posture between the first entity and the second entity can be effectively and accurately determined.

[0085] In order to improve the accuracy of the feature point matching process, in a possible implementation method of an embodiment of the present application, after obtaining the first matched feature point pair, a RANSAC (Random Sample Consensus) algorithm can be used to perform geometric consistency verification on the obtained first matched feature point pair. This can effectively remove erroneous matches in the feature point matching process and improve the accuracy and robustness of subsequent tasks.

[0086] It is understandable that there may be a situation where the similarity is not greater than the first set threshold. In this case, it indicates that the first entity and the second entity in any two entities have not observed similar scenes. Therefore, in a possible implementation method of the embodiment of the present disclosure, when the similarity is not greater than the first set threshold, it can be determined from multiple entities whether there is a target entity; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold; when there is a target entity among multiple entities, the first relative posture between the first entity and the second entity can be determined based on the conversion relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the conversion relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

[0087] It should be noted that the method for obtaining the similarity between the sparse map of the target entity and the sparse map of the first entity, and the method for obtaining the similarity between the sparse map of the target entity and the sparse map of the second entity are similar to the method for obtaining the similarity between the sparse map of the first entity and the sparse map of the second entity in step S204, and are not repeated here.

[0088] In an embodiment of the present application, when there is a target entity among multiple entities, the first relative posture between the first entity and the second entity can be determined based on the conversion relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the conversion relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity.

[0089] As an example, when there is a target entity among multiple entities, it is assumed that the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity is as shown in the following formula: ; (3) Among them, P1 represents the spatial coordinates of the feature point of the first entity in the entity coordinate system of the first entity, which is a vector with 3 rows and 1 column; P is the spatial coordinates of the feature point of the target entity in the entity coordinate system of the target entity, which is a vector with 3 rows and 1 column; R is a rotation matrix with 3 rows and 3 columns, which represents the rotation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; T is a translation vector with 3 rows and 1 column, which represents the translation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; The transformation relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity is shown in the following formula: ; (4) Among them, P2 represents the spatial coordinates of the feature point of the second entity in the entity coordinate system of the second entity, which is a vector with 3 rows and 1 column; P is the spatial coordinates of the feature point of the target entity in the entity coordinate system of the target entity, which is a vector with 3 rows and 1 column; R' is a rotation matrix with 3 rows and 3 columns, which represents the rotation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; T' is a translation vector with 3 rows and 1 column, which represents the translation operation from the entity coordinate system of the target entity to the entity coordinate system of the first entity; Based on the conversion relationship between the entity coordinate system of the first entity and the entity coordinate system of the target entity, and the conversion relationship between the entity coordinate system of the second entity and the entity coordinate system of the target entity, the first relative pose between the first entity and the second entity can be determined according to the following formula: ; (5) The first relative posture between the first entity and the second entity is: the rotation matrix of the second entity relative to the first entity is R'R, and the translation vector is (R'T+T').

[0090] Therefore, when the similarity between the sparse map of the first entity and the sparse map of the second entity is not greater than the first set threshold, the first relative posture between the first entity and the second entity can be effectively and accurately determined with the help of the target entity.

[0091] Step S206: analyzing the first relative posture between the first entity and the second entity to obtain the relative position between the first entity and the second entity.

[0092] It should be noted that the explanation of the relative position in step S104 is also applicable to this embodiment and will not be repeated here.

[0093] Specifically, in the case where the first relative posture includes a translation vector and a rotation matrix, the spatial position of the first entity in the physical coordinate system of the second entity and / or the spatial position of the first entity in the physical coordinate system of the second entity, as well as the relative distance between the first entity and the second entity can be obtained based on the translation vector in the first relative posture between the first entity and the second entity; the rotation angle of the first entity around the corresponding coordinate axis of the physical coordinate system of the second entity can be obtained based on the rotation matrix in the first relative posture between the first entity and the second entity.

[0094] The method for determining the relative position of multiple entities in the embodiment of the present application obtains the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities; in response to the similarity being greater than a first set threshold, the sparse map of the first entity and the sparse map of the second entity are aligned to obtain a first relative posture between the first entity and the second entity; the first relative posture between the first entity and the second entity is analyzed to obtain the relative position between the first entity and the second entity. Thus, by calculating the similarity between the sparse maps of any two entities and aligning the sparse maps when the similarity meets certain conditions, the relative position between the entities is determined, thereby improving the accuracy and efficiency of determining the relative position.

[0095] When the image data corresponding to each entity includes multiple continuous image frames and each image frame has corresponding multiple feature points, in order to clearly illustrate how a sparse map of the entity is constructed based on the multiple feature points corresponding to any entity in the above embodiments of the present application, the present application also proposes a method for determining the relative positions of multiple entities.

[0096] Figure 4 A schematic flow chart of a method for determining relative positions of multiple entities provided in another embodiment of the present application.

[0097] like Figure 4 As shown, the method comprises the following steps: Step S401 , obtaining image data of the environment in which the corresponding entity is located through a visual sensor equipped by each entity among multiple entities.

[0098] Step S402: for any entity, extract feature points from the image data corresponding to the entity to obtain a plurality of feature points corresponding to the entity.

[0099] It should be noted that the execution process of step S401 to step S402 can refer to the relevant description in any embodiment of the present application and will not be repeated here.

[0100] Step S403 , sorting multiple consecutive image frames corresponding to the entity in the order of their acquisition time to obtain an image sorting sequence of the entity.

[0101] Step S404 , determining the camera coordinate system of the image frame with the sequence number of the first in the image sorting sequence of the entity collected by the visual sensor equipped by the entity as the entity coordinate system of the entity.

[0102] That is to say, when the visual sensor equipped by the entity collects the image frame with the sequence number 1 in the image sorting sequence of the entity, the camera coordinate system coincides with the entity coordinate system of the entity.

[0103] Step S405 , according to the image sorting sequence of the entity, sequentially determine the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity.

[0104] Alternatively, if Figure 5 As shown, the following steps can be used to determine the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity: Step S4051 , for any feature point of the image frame with the sequence number being the first in the image sorting sequence of the entity, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the image coordinates of the feature point.

[0105] The image coordinates may be used to indicate the coordinate position of the corresponding feature point in the corresponding image frame.

[0106] In the embodiment of the present application, the spatial coordinates of the feature points in the corresponding entity coordinate system can be determined based on the image coordinates of the feature points. Specifically, the feature points can be triangulated based on the image coordinates of the feature points and the internal parameters of the visual sensor equipped by the entity (such as focal length, principal point coordinates, etc.) to obtain the spatial coordinates of the feature points in the corresponding entity coordinate system.

[0107] Step S4052: for the i+1th image frame in the image sorting sequence corresponding to the entity, multiple feature points of the i-th image frame are matched with multiple feature points of the i+1th image frame to obtain a matched second feature point pair.

[0108] Here, i can be a positive integer not equal to 0.

[0109] It should be noted that the present application does not limit the number of second feature point pairs.

[0110] As an example, for the i+1th image frame in the image sorting sequence corresponding to the entity, a matching algorithm such as FLANN and BFMatcher can be used to match multiple feature points of the i-th image frame with multiple feature points of the i+1th image frame to obtain a matched second feature point pair.

[0111] In the feature point matching process, due to factors such as image noise, illumination changes, and perspective differences, some erroneous matching pairs are often generated, and these erroneous matching pairs will have a negative impact on subsequent data applications. In order to improve the accuracy of the feature point matching process, in a possible implementation of an embodiment of the present application, after obtaining the matched second feature point pair, the RANSAC (Random Sample Consensus) algorithm can be used to perform geometric consistency verification on the obtained matched second feature point pair, thereby effectively removing erroneous matches in the feature point matching process and improving the accuracy and robustness of subsequent tasks.

[0112] Step S4053: Based on the matched second feature point pair, determine the second relative posture of the visual sensor equipped by the entity when capturing the i+1th image frame relative to when capturing the i-th image frame.

[0113] The second relative posture may include a translation vector and a rotation matrix.

[0114] In an embodiment of the present application, the second relative posture of the visual sensor equipped by the entity when collecting the i+1th image frame relative to the i-th image frame can be determined based on the second feature point pair matched between the i-th image frame and the i+1th image frame. Specifically, the second relative posture of the visual sensor equipped by the entity when collecting the i+1th image frame relative to the i-th image frame can be determined based on the spatial coordinates of the feature points belonging to the i-th image frame in the second feature point pair matched between the i-th image frame and the i+1th image frame and the image coordinates of the feature points belonging to the i+1th image frame. For example, the second relative posture of the visual sensor equipped by the entity when collecting the i+1th image frame relative to the i-th image frame can be determined based on the spatial coordinates of the feature points belonging to the i-th image frame in the second feature point pair matched between the i-th image frame and the i+1th image frame and the image coordinates of the feature points belonging to the i+1th image frame, using a posture estimation algorithm (such as a PnP (Perspective-n-Points) algorithm).

[0115] Step S4054: for any feature point of the (i+1)th image frame, based on the second relative posture and the image coordinates of the feature point, determine the spatial coordinates of the feature point in the corresponding physical coordinate system.

[0116] As an example, first, for any feature point of the i+1th image, based on the image coordinates of the feature point, the spatial coordinates of the feature point in the camera coordinate system when the visual sensor equipped by the entity collects the i+1th image frame are determined. For example, the feature point can be triangulated based on the image coordinates of the feature point and the internal parameters of the visual sensor equipped by the entity (such as focal length, principal point coordinates, etc.) to obtain the spatial coordinates of the feature point in the camera coordinate system when the visual sensor equipped by the entity collects the i+1th image frame; secondly, based on the second relative posture of the visual sensor equipped by the entity when collecting the i+1th image frame relative to the i-th image frame, the estimated posture of the camera coordinate system of the visual sensor equipped by the entity when collecting the i+1th image frame relative to the entity coordinate system of the entity can be determined; finally, based on the estimated posture and the internal parameters of the visual sensor equipped by the entity (such as focal length, principal point coordinates, etc.), the spatial coordinates of the feature point in the camera coordinate system when the visual sensor equipped by the entity collects the i+1th image frame are converted into spatial coordinates in the entity coordinate system.

[0117] Among them, when determining the estimated pose of the camera coordinate system of the visual sensor equipped by the entity when capturing the i+1th image frame relative to the physical coordinate system of the entity based on the second relative pose of the visual sensor equipped by the entity when capturing the i+1th image frame relative to the i-th image frame, for example, the estimated pose of the camera coordinate system of the visual sensor equipped by the entity when capturing the i+1th image frame relative to the physical coordinate system of the entity can be determined based on the second relative pose of the visual sensor equipped by the entity when capturing the i+1th image frame relative to the i-th image frame, and the second relative pose corresponding to any adjacent image frame before the visual sensor captures the i+1th image frame.

[0118] For example, assuming that i is 3, the second relative position of the visual sensor equipped with the entity when collecting the fourth image frame relative to the second relative position when collecting the third image frame is: the translation vector T 34 , rotation matrix R 34 ; The second relative position corresponding to any adjacent image frame before the fourth image frame collected by the visual sensor includes: the second relative position of the visual sensor when collecting the third image frame relative to the second image frame collected is: the translation vector T 23 , rotation matrix R 23 , the second relative position of the visual sensor when collecting the second image frame relative to the first image frame is: the translation vector T 12 , rotation matrix R 12 ; Then the estimated pose of the camera coordinate system of the visual sensor equipped with the entity when collecting the i+1th image frame relative to the entity coordinate system of the entity is: rotation matrix (R 34 R 23 R12 ), translation vector (R 34 R 23 T 12 +R 34 T 23 +T 34 ). Thus, the estimated pose of the camera coordinate system of the visual sensor equipped with the entity relative to the entity coordinate system when capturing the i+1th image frame can be effectively determined.

[0119] In summary, the spatial coordinates of each feature point of the corresponding image frame can be determined frame by frame according to the acquisition time of multiple continuous image frames corresponding to the entity.

[0120] Step S406: construct a sparse map of the entity based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system.

[0121] In an embodiment of the present application, each feature point corresponding to the entity may be fused based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system, thereby obtaining a sparse map of the entity.

[0122] Step S407: aligning the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between any two entities.

[0123] It should be noted that the execution process of step S407 can refer to the relevant description in any embodiment of the present application and will not be repeated here.

[0124] The method for determining the relative position of multiple entities in the embodiment of the present application obtains an image sorting sequence of the entity by sorting the multiple continuous image frames corresponding to the entity in the order of the acquisition time; determines the camera coordinate system of the image frame with the sequence number 1 in the image sorting sequence of the entity acquired by the visual sensor equipped by the entity as the entity coordinate system of the entity; determines the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity in turn according to the image sorting sequence of the entity; and constructs a sparse map of the entity based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system. Thus, the construction of the sparse map of each entity is effectively realized, providing key data support for the subsequent determination of the relative position between multiple entities.

[0125] In order to clearly illustrate the method for determining relative positions of multiple entities of the present application, a detailed description is given below with reference to examples.

[0126] As an example, the method for determining relative positions of multiple entities may include the following steps: 1. For any of the multiple entities, real-time SLAM (Simultaneous Localization and Mapping) is performed while the entity is running, specifically: 1.1 Data Collection Collecting image data of the environment in which the entity is located through at least one visual sensor equipped by the entity; Among them, the visual sensor can be a monocular camera, a binocular camera or an RGB-D camera, etc., and this application does not impose any restrictions on this.

[0127] 1.2 Feature Point Extraction Extract feature points from image data to obtain multiple feature points corresponding to the entity. For example, SIFT, SURF, ORB and other algorithms can be used to detect or identify feature points of image data, and the detected or identified feature points are described to obtain feature point descriptors of the feature points. The feature points may be, but are not limited to, corner points, edges, and the like.

[0128] 1.3 Feature Point Matching When the image data includes multiple continuous image frames, feature point matching can be performed between adjacent image frames in the multiple continuous image frames to obtain matching second feature point pairs. For example, matching algorithms such as FLANN and BFMatcher can be used to obtain matching second feature point pairs between adjacent image frames.

[0129] It should be noted that false matches may occur during the feature point matching process. In order to improve the matching accuracy, the RANSAC (random sampling consensus) algorithm can be used to verify the geometric consistency of the second feature point pair obtained to remove false matches.

[0130] 1.4 Pose Estimation Based on the matching second feature point pairs between adjacent image frames, the pose estimation algorithm (such as the PnP algorithm) is used to estimate the camera pose, and the second relative pose of the visual sensor when collecting the corresponding adjacent image frames is obtained. And through the multi-view set relationship, the estimated pose of the visual sensor equipped by the calculation entity when collecting different image frames.

[0131] 1.5 Sparse map construction Based on the estimated pose, the feature points are back-projected to obtain the spatial coordinates of the feature points in three-dimensional space. Based on the spatial coordinates of the feature points corresponding to the entity, the feature points corresponding to the entity are fused to obtain a sparse map of the entity.

[0132] Optionally, a bundle adjustment (BA) algorithm or a filtering method (such as EKF (Extended Kalman Filter), UKF (Unscented Kalman Filter), etc.) can be used to globally optimize the estimated pose and spatial coordinates of feature points to reduce errors and improve map accuracy.

[0133] 2. Align the sparse maps of any two entities to determine the relative position between any two entities. Specifically: 2.1 Feature Point Matching Feature point matching is performed on the feature points of the sparse maps of any two entities to obtain a first matched feature point pair. For example, a matching algorithm such as FLANN or BFMatcher may be used to obtain a first matched feature point pair.

[0134] It should be noted that in step 1.2, after the feature points of the entity are extracted, the feature descriptors of the feature points of the entity can be saved to facilitate subsequent data applications, such as feature point matching.

[0135] 2.2 Determination of the relative position between any two entities Determine the similarity between the sparse maps of any two entities; when the similarity is greater than a first set threshold, it is determined that the visual sensors corresponding to the two entities have observed similar scenes and a loop can be formed. Thus, based on the matched first feature point pairs between the two entities, the transformation relationship between the world coordinate systems (referred to as entity coordinate systems in this application) of the two entities can be determined, and based on the transformation relationship, the relative positions of the two entities can be determined.

[0136] Among them, based on the conversion relationship, the coordinates of one of any two entities can be converted to the world coordinate system of the other entity, thereby realizing the alignment of the sparse maps of the two entities.

[0137] Optionally, the aligned sparse maps may be fused to obtain a merged map corresponding to the two entities, and so on to achieve map fusion between multiple entities, which will not be elaborated here.

[0138] It is understandable that as the SLAM algorithms of each entity continue to run, the relative positions between entities can be updated and dynamically adjusted in real time, and the continuous operation of the SLAM process and map fusion algorithm can ensure that the complete merged map can be synchronously updated to each continuously running entity, and the new map areas explored by each continuously running entity can also be continuously passed to other entities by merging maps. In this way, the three-dimensional map reconstruction of a single entity can be expanded to the three-dimensional map reconstruction and fusion of multiple entities. As long as there is a common view or a common view area between any two entities in the multiple entities, the maps between the multiple entities can be fused by fusing map feature points, and the relative positions between the multiple entities can be determined. This method does not require the introduction of external markers or base stations, reduces deployment costs and usage costs, is easy to expand to multiple devices, and has strong environmental adaptability.

[0139] Optionally, for any entity, the entity can also be equipped with an IMU (Inertial Measurement Unit) or other auxiliary sensors to provide additional position information and motion data.

[0140] The multi-entity relative position determination method can be applied to a multi-entity relative position determination system, wherein the multi-entity relative position determination system comprises: 1. Visual sensor, used to capture images in the environment and generate image data.

[0141] 2. IMU (optional), used to provide auxiliary positioning information, including acceleration and angular velocity data; it should be noted that IMU can help improve the positioning accuracy of the system in the case of rapid movement or visual sensor failure.

[0142] 3. SLAM algorithm unit, used to process visual sensor data and IMU data for real-time 3D map reconstruction and positioning. The SLAM algorithm unit includes front-end and back-end processing modules; the front-end is responsible for feature point extraction, matching and preliminary pose estimation, and the back-end is responsible for global optimization and map construction.

[0143] 4. Feature point extraction and matching module, used to extract feature points from image data and perform feature point matching on the feature points.

[0144] 5. Position optimization module, used to optimize and correct the estimated pose and constructed map to ensure the accuracy of the relative position.

[0145] 6. A map construction module is used to construct a sparse three-dimensional point cloud map (referred to as a sparse map in this application) based on the pose and feature points obtained from the SLAM algorithm unit, and to update and maintain the map in real time, and to perform global consistency optimization by detecting loop closures.

[0146] 7. A map fusion module is used to fuse the sparse three-dimensional point cloud maps corresponding to multiple entities. Specifically, by matching the feature points in the sparse three-dimensional point cloud maps of different entities, the coordinate transformation matrix is ​​calculated, and based on the coordinate transformation matrix, the map coordinate system of one entity is converted to the map coordinate system of another entity, thereby realizing the unification of multiple entity maps.

[0147] 8. Transmission and synchronization module, used to exchange maps and location information between multiple entities through wired or wireless communication to ensure data synchronization of each entity. It should be noted that the transmission and synchronization module supports real-time map fusion and location update to ensure the global consistency and real-time performance of the system.

[0148] The method for determining the relative positions of multiple entities in the present application has at least the following advantages: 1. No external marking required: It is free from the reliance on external calibration objects and has a wider range of applications; 2. Strong environmental adaptability: It has strong adaptive capability and is suitable for complex and dynamic environments. It is not restricted by the installation and maintenance of calibration objects, nor is it affected by occlusion. 3. Quick addition of new devices: There is no need for complicated configuration and calibration processes when adding new devices to the corresponding system, which helps ensure seamless integration and interoperability between new devices and existing systems.

[0149] The inventor of the present application has verified the feasibility of the method for determining the relative positions of multiple entities through experiments. Multiple entities equipped with visual sensors have been three-dimensionally reconstructed and fused into maps in different environments, and the relative positions between the multiple entities have been accurately determined. The experimental results show that this method can quickly and accurately determine the relative positions between multiple entities, and maintain high real-time performance and stability in a dynamic environment.

[0150] In order to implement the above embodiment, the present application also proposes a device for determining relative positions of multiple entities.

[0151] Figure 6 A schematic diagram of the structure of a device for determining relative positions of multiple entities provided in another embodiment of the present application.

[0152] like Figure 6 As shown, the device 600 for determining relative positions of multiple entities includes: an acquisition module 610 , an extraction module 620 , a construction module 630 and an alignment module 640 .

[0153] The acquisition module 610 is used to acquire image data of the environment where the corresponding entity is located through a visual sensor equipped by each entity among the multiple entities.

[0154] The extraction module 620 is used to extract feature points from the image data corresponding to any entity to obtain a plurality of feature points corresponding to the entity.

[0155] The construction module 630 is used to construct a sparse map of the entity based on multiple feature points corresponding to the entity.

[0156] The alignment module 640 is used to align the sparse maps corresponding to any two entities among the multiple entities to determine the relative positions between any two entities.

[0157] Furthermore, in a possible implementation of an embodiment of the present application, any two entities include a first entity and a second entity; an alignment module 640 is used to: obtain the similarity between a sparse map of the first entity and a sparse map of the second entity in any two entities; in response to the similarity being greater than a first set threshold, align the sparse map of the first entity and the sparse map of the second entity to obtain a first relative posture between the first entity and the second entity; and parse the first relative posture between the first entity and the second entity to obtain a relative position between the first entity and the second entity in any two entities.

[0158] Furthermore, in a possible implementation of an embodiment of the present application, the alignment module 640 is used to: determine a first number of feature points in the sparse map of the first entity that match the feature points in the sparse map of the second entity; determine a first coefficient based on the first number and the total number of feature points in the sparse map of the first entity; determine a second number of feature points in the sparse map of the second entity that match the feature points in the sparse map of the first entity; determine a second coefficient based on the second number and the total number of feature points in the sparse map of the second entity; and determine the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities based on the first coefficient and / or the second coefficient.

[0159] Furthermore, in a possible implementation of an embodiment of the present application, the alignment module 640 is used to: match each feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity to obtain a matched first feature point pair; based on the matched first feature point pair, determine the transformation relationship between the entity coordinate system of the first entity and the entity coordinate system of the second entity; based on the transformation relationship, determine a first relative posture between the first entity and the second entity.

[0160] Furthermore, in a possible implementation of an embodiment of the present application, the device also includes a first determination module and a second determination module, the first determination module being used to: in response to the similarity not being greater than a first set threshold, determine whether there is a target entity from multiple entities; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold; the second determination module being used to: in response to the presence of the target entity among multiple entities, determine a first relative pose between the first entity and the second entity based on a conversion relationship between a physical coordinate system of the first entity and a physical coordinate system of the target entity, and a conversion relationship between a physical coordinate system of the second entity and a physical coordinate system of the target entity.

[0161] Furthermore, in a possible implementation of an embodiment of the present application, the image data includes multiple continuous image frames, and correspondingly, each image frame has multiple corresponding feature points; a construction module 630 is used to: sort the multiple continuous image frames corresponding to the entity in the order of acquisition time to obtain an image sorting sequence of the entity; determine the camera coordinate system when the visual sensor equipped by the entity acquires the image frame with sequence number 1 in the image sorting sequence of the entity as the entity coordinate system of the entity; determine the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system in turn according to the image sorting sequence of the entity; and construct a sparse map of the entity based on the spatial coordinates of each feature point corresponding to the entity in the corresponding entity coordinate system.

[0162] Furthermore, in a possible implementation of an embodiment of the present application, a construction module 630 is used to: for any feature point of an image frame with a serial number of 1 in an image sorting sequence of an entity, determine the spatial coordinates of the feature point in a corresponding entity coordinate system based on the image coordinates of the feature point; for an i+1th image frame in an image sorting sequence corresponding to the entity, match multiple feature points of the i-th image frame with multiple feature points of the i+1th image frame to obtain a matched second feature point pair; wherein i is a positive integer not equal to 0; based on the matched second feature point pair, determine the second relative posture of the visual sensor equipped by the entity when collecting the i+1th image frame relative to when collecting the i-th image frame; for any feature point of the i+1th image frame, determine the spatial coordinates of the feature point in the corresponding entity coordinate system based on the second relative posture and the image coordinates of the feature point.

[0163] It should be noted that the above explanation of the embodiment of the method for determining relative positions of multiple entities is also applicable to the device for determining relative positions of multiple entities of this embodiment, and will not be repeated here.

[0164] In summary, the device for determining the relative positions of multiple entities in the embodiment of the present application can automatically determine the relative positions between multiple entities by constructing a sparse map of each entity in the multiple entities and aligning the sparse maps. It does not rely on external calibration objects and is suitable for various complex and dynamic environments. It has strong adaptability and high flexibility. Since there is no need to predefine the pattern or features of the calibration object, the complexity of algorithm development and adaptation is reduced, thereby reducing the adaptation cost and time. Even if some feature points are blocked or the environment changes, the sparse map alignment and positioning can still be performed through the remaining feature points, thereby improving the robustness, accuracy and reliability of relative position determination.

[0165] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method for determining the relative positions of multiple entities provided in the above embodiments.

[0166] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method for determining the relative positions of multiple entities provided in the above embodiments.

[0167] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the method for determining relative positions of multiple entities provided in the above embodiments.

[0168] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0169] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0170] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0171] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0172] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to control or imply relative importance or implicitly indicate the number of technical features controlled. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0173] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0174] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0175] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0176] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0178] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for determining relative positions of multiple entities, characterized in that: The method comprises: Acquire image data of an environment in which the corresponding entity is located through a visual sensor equipped by each of the multiple entities; For any of the entities, extract feature points from the image data corresponding to the entity to obtain a plurality of feature points corresponding to the entity; Based on the multiple feature points corresponding to the entity, construct a sparse map of the entity; The sparse maps corresponding to any two of the multiple entities are aligned to determine the relative positions between the any two entities.

2. The method according to claim 1, characterized in that The any two entities include a first entity and a second entity; The aligning the sparse maps corresponding to any two of the multiple entities to determine the relative positions between the any two entities includes: Obtaining a similarity between a sparse map of the first entity and a sparse map of the second entity in any two entities; In response to the similarity being greater than a first set threshold, aligning the sparse map of the first entity and the sparse map of the second entity to obtain a first relative pose between the first entity and the second entity; A first relative position between the first entity and the second entity is analyzed to obtain a relative position between any two entities.

3. The method according to claim 2, characterized in that The obtaining the similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities includes: determining a first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity; determining a first coefficient according to the first number and a total number of feature points in the sparse map of the first entity; determining a second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity; determining a second coefficient according to the second number and a total number of feature points in the sparse map of the second entity; Based on the first coefficient and / or the second coefficient, a similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities is determined.

4. The method according to claim 2, characterized in that: The aligning the sparse map of the first entity and the sparse map of the second entity to obtain a first relative posture between the first entity and the second entity includes: Matching each feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity to obtain a matched first feature point pair; Determining a conversion relationship between a physical coordinate system of the first entity and a physical coordinate system of the second entity based on the matched first feature point pair; Based on the transformation relationship, a first relative position and posture between the first entity and the second entity is determined.

5. The method according to claim 4, characterized in that The method further comprises: In response to the similarity not being greater than the first set threshold, determining whether a target entity exists from the plurality of entities; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold; In response to the existence of the target entity among the multiple entities, a first relative posture between the first entity and the second entity is determined based on a conversion relationship between a physical coordinate system of the first entity and a physical coordinate system of the target entity, and a conversion relationship between a physical coordinate system of the second entity and a physical coordinate system of the target entity.

6. The method according to any one of claims 1 to 5, characterized in that: The image data includes a plurality of continuous image frames, and correspondingly, each of the image frames has a corresponding plurality of feature points; The step of constructing a sparse map corresponding to the entity based on the plurality of feature points corresponding to the entity includes: Sorting multiple consecutive image frames corresponding to the entity in chronological order of acquisition time to obtain an image sorting sequence of the entity; Determine the camera coordinate system when the visual sensor equipped by the entity captures the image frame with sequence number 1 in the image sorting sequence of the entity as the entity coordinate system of the entity; According to the image sorting sequence of the entity, sequentially determining the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity; Based on the spatial coordinates of each of the feature points corresponding to the entity in the corresponding entity coordinate system, a sparse map of the entity is constructed.

7. The method according to claim 6, characterized in that The step of sequentially determining the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity according to the image sorting sequence of the entity includes: For any of the feature points of the image frame with a sequence number of 1 in the image sorting sequence of the entity, determining the spatial coordinates of the feature point in the corresponding entity coordinate system based on the image coordinates of the feature point; For the i+1th image frame in the image sorting sequence corresponding to the entity, multiple feature points of the i-th image frame are matched with multiple feature points of the i+1th image frame to obtain a matched second feature point pair; wherein i is a positive integer not equal to 0; Based on the matched second feature point pair, determining a second relative posture of the visual sensor equipped by the entity when capturing the (i+1)th image frame relative to when capturing the (i)th image frame; For any of the feature points in the (i+1)th image frame, based on the second relative posture and the image coordinates of the feature point, determine the spatial coordinates of the feature point in the corresponding physical coordinate system.

8. A device for determining relative positions of multiple entities, characterized in that: The device comprises: An acquisition module, used for acquiring image data of an environment in which a corresponding entity is located through a visual sensor equipped by each of the multiple entities; An extraction module, used for extracting feature points from image data corresponding to any of the entities to obtain a plurality of feature points corresponding to the entity; A construction module, configured to construct a sparse map of the entity based on a plurality of feature points corresponding to the entity; The alignment module is used to align the sparse maps corresponding to any two of the multiple entities to determine the relative positions between the any two entities.

9. The device according to claim 8, characterized in that The any two entities include a first entity and a second entity; the alignment module is used to: Obtaining a similarity between a sparse map of the first entity and a sparse map of the second entity in any two entities; In response to the similarity being greater than a first set threshold, aligning the sparse map of the first entity and the sparse map of the second entity to obtain a first relative pose between the first entity and the second entity; A first relative position between the first entity and the second entity is analyzed to obtain a relative position between the first entity and the second entity in any two entities.

10. The device according to claim 9, characterized in that The alignment module is used to: determining a first number of feature points in the sparse map of the first entity that match feature points in the sparse map of the second entity; determining a first coefficient according to the first number and a total number of feature points in the sparse map of the first entity; determining a second number of feature points in the sparse map of the second entity that match feature points in the sparse map of the first entity; determining a second coefficient according to the second number and a total number of feature points in the sparse map of the second entity; Based on the first coefficient and / or the second coefficient, a similarity between the sparse map of the first entity and the sparse map of the second entity in any two entities is determined.

11. The device according to claim 9, characterized in that The alignment module is used to: Matching each feature point in the sparse map of the first entity with each feature point in the sparse map of the second entity to obtain a matched first feature point pair; Determining a conversion relationship between a physical coordinate system of the first entity and a physical coordinate system of the second entity based on the matched first feature point pair; Based on the transformation relationship, a first relative position and posture between the first entity and the second entity is determined.

12. The device according to claim 11, characterized in that The device also includes: a first determination module, configured to determine whether a target entity exists from the plurality of entities in response to the similarity not being greater than the first set threshold; wherein the similarity between the sparse map of the target entity and the sparse map of the first entity is greater than the first set threshold, and the similarity between the sparse map of the target entity and the sparse map of the second entity is greater than the first set threshold; A second determination module is used to determine a first relative posture between the first entity and the second entity based on a transformation relationship between a physical coordinate system of the first entity and a physical coordinate system of the target entity, and a transformation relationship between a physical coordinate system of the second entity and a physical coordinate system of the target entity in response to the existence of the target entity among the multiple entities.

13. The device according to any one of claims 8 to 12, characterized in that: The image data includes a plurality of continuous image frames, and correspondingly, each of the image frames has a corresponding plurality of feature points; The building blocks are used to: Sorting multiple consecutive image frames corresponding to the entity in chronological order of acquisition time to obtain an image sorting sequence of the entity; Determine the camera coordinate system when the visual sensor equipped by the entity captures the image frame with sequence number 1 in the image sorting sequence of the entity as the entity coordinate system of the entity; According to the image sorting sequence of the entity, sequentially determining the spatial coordinates of each feature point corresponding to each image frame in the entity coordinate system of the entity; Based on the spatial coordinates of each of the feature points corresponding to the entity in the corresponding entity coordinate system, a sparse map of the entity is constructed.

14. The device according to claim 13, characterized in that The building blocks are used to: For any of the feature points of the image frame with a sequence number of 1 in the image sorting sequence of the entity, determining the spatial coordinates of the feature point in the corresponding entity coordinate system based on the image coordinates of the feature point; For the i+1th image frame in the image sorting sequence corresponding to the entity, multiple feature points of the i-th image frame are matched with multiple feature points of the i+1th image frame to obtain a matched second feature point pair; wherein i is a positive integer not equal to 0; Based on the matched second feature point pair, determining a second relative posture of the visual sensor equipped by the entity when capturing the (i+1)th image frame relative to when capturing the (i)th image frame; For any of the feature points in the (i+1)th image frame, based on the second relative posture and the image coordinates of the feature point, determine the spatial coordinates of the feature point in the corresponding physical coordinate system.

15. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

17. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images