Multi-view autonomous cooperative positioning method and system

Abstract

The invention provides a multi-view autonomous cooperative positioning method and system. The method comprises the following steps: determining a plurality of local maps and a plurality of key frame information of an environment where a plurality of agents are located based on a preset synchronous positioning and mapping algorithm; the plurality of intelligent agents are used for acquiring local environment information of multiple viewpoints from different positions and visual angles through carried visual sensors; the multi-view local environment information corresponds to a plurality of local maps and a plurality of key frame information; determining a constraint relationship among the plurality of local maps according to the information of the plurality of key frames, and determining a local map coordinate system transformation matrix according to the constraint relationship; the constraint relation comprises pose constraints of adjacent key frames; and fusing the plurality of local maps according to the local map coordinate system transformation matrix to obtain a global map, and determining global positioning information according to the global map. The method can adapt to a complex environment, and realizes high-precision and high-robustness unmanned aerial vehicle cluster positioning.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) positioning technology, and in particular to a multi-viewpoint autonomous collaborative positioning method and system. Background Art

[0002] With the rapid development of drone technology, drone swarms are increasingly being used in military, civilian, and commercial applications. Drone swarm operations can achieve efficient mission coordination, enhance robustness, and expand mission coverage. However, drone swarm positioning technology, as one of its core key technologies, still faces numerous challenges.

[0003] Currently, drone positioning primarily relies on the Global Positioning System (GPS) or Inertial Navigation System (INS), but these methods have numerous limitations in complex environments. For example, GPS signals are easily blocked or interfered with in complex environments such as urban canyons, mountainous areas, and indoors, resulting in a significant decrease in positioning accuracy. When performing missions, drone swarms require precise knowledge of the relative positions and attitudes of each drone within the swarm to achieve coordinated control. However, a single positioning method is unable to meet this requirement. Therefore, developing a high-precision, robust drone swarm positioning technology that can adapt to complex environments remains a pressing challenge in the field of drone technology. Summary of the Invention

[0004] The present invention provides a multi-viewpoint autonomous collaborative positioning method and system, which can adapt to complex environments and achieve high-precision and high-robustness UAV cluster positioning.

[0005] The present invention provides a multi-viewpoint autonomous collaborative positioning method, comprising: determining multiple local maps and multiple key frame information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; the multiple intelligent agents are used to obtain local environmental information of multiple viewpoints from different positions and perspectives through the visual sensors they carry; the local environmental information of the multiple viewpoints corresponds to the multiple local maps and the multiple key frame information; determining a constraint relationship between the multiple local maps based on the multiple key frame information, and determining a local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes posture constraints of adjacent key frames; and fusing the multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information based on the global map.

[0006] According to a multi-viewpoint autonomous collaborative positioning method provided by the present invention, the visual sensor is a camera, and the local environment information of the multiple viewpoints is the local environment images of the multiple viewpoints; the multiple local maps and multiple key frame information of the environment where the multiple intelligent agents are located are determined based on a preset synchronous positioning and mapping algorithm, including: obtaining the local environment images of the multiple viewpoints collected by the camera; obtaining the camera pose information of the image key frame based on the local environment images of the multiple viewpoints and the intrinsic parameter matrix of the camera; and constructing multiple local maps of the environment where the multiple intelligent agents are located based on the camera pose information of the image key frame and the local environment images of the multiple viewpoints.

[0007] According to a multi-viewpoint autonomous collaborative positioning method provided by the present invention, the constraint relationship also includes a loop detection constraint; the constraint relationship between the multiple local maps is determined based on the multiple key frame information, including: determining a pose graph based on the camera pose information of the multiple image key frames; the nodes of the pose graph represent the camera poses of the key frames; the edges of the pose graph represent the relative pose constraints between the connected key frames; based on the camera pose information of the multiple image key frames, a loop detection algorithm is used to take the constraints between the current frame and the historical frames as the loop detection constraints.

[0008] According to a multi-viewpoint autonomous collaborative positioning method provided by the present invention, a binary selection algorithm is used to filter key frames from frames that are a preset number of frames away from a current frame.

[0009] According to a multi-viewpoint autonomous collaborative positioning method provided by the present invention, determining the local map coordinate system transformation matrix based on the constraint relationship includes: using an extreme geometry algorithm to calculate the posture transformation relationship between the image key frames based on the constraint relationship to obtain the local map coordinate system transformation matrix.

[0010] According to a multi-viewpoint autonomous collaborative positioning method provided by the present invention, the method also includes: in a general graph optimization framework, using map points and camera poses as nodes and reprojection errors as edges, and using a global optimization algorithm to globally optimize the global map; the optimization goal of the global optimization algorithm is to minimize the reprojection error; the reprojection error is the difference between the projected position of the map point in the camera and the actual observed position.

[0011] The present invention also provides a multi-viewpoint autonomous collaborative positioning system, including: a local map determination module, which is used to determine multiple local maps and multiple key frame information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; multiple intelligent agents are used to obtain multi-viewpoint local environment information from different positions and perspectives through the carried visual sensors; the multi-viewpoint local environment information corresponds to multiple local maps and multiple key frame information; a constraint relationship determination module, which is used to determine the constraint relationship between multiple local maps based on multiple key frame information, and determine the local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes the posture constraints of adjacent key frames; a global positioning module, which is used to fuse multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information based on the global map.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-viewpoint autonomous collaborative positioning method as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-viewpoint autonomous collaborative positioning methods described above.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the multi-viewpoint autonomous collaborative positioning methods described above.

[0015] The present invention provides a multi-viewpoint autonomous collaborative positioning method and system, which includes: determining multiple local maps and multiple keyframe information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; the multiple intelligent agents are configured to obtain multi-viewpoint local environment information from different positions and perspectives using onboard visual sensors; the multi-viewpoint local environment information corresponds to the multiple local maps and multiple keyframe information; determining the constraint relationships between the multiple local maps based on the multiple keyframe information, and determining the local map coordinate system transformation matrix based on the constraint relationships; the constraint relationships include position constraints of adjacent keyframes; and fusing the multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, thereby determining global positioning information based on the global map. The present invention can adapt to complex environments and achieve high-precision and high-robustness drone cluster positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a flow chart of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0018] Figure 2 This is a schematic diagram of the principle of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0019] Figure 3 This is a specific flow chart of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0020] Figure 4 It is a structural diagram of a multi-viewpoint autonomous collaborative positioning system provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0023] As drone technology matures, drone swarms are finding increasing applications. In the military, drone swarms can be used for reconnaissance, strikes, and communications relay; in the civilian sector, they are used for environmental monitoring, disaster relief, and traffic control; and in the commercial sector, they encompass logistics, agriculture, electricity, surveying and mapping, and film and television production. Leveraging the collaborative capabilities of multiple drones, drone swarms are applicable in numerous scenarios and hold broad application prospects. Effectively achieving positioning in drone swarms has become a research hotspot.

[0024] Please refer to Figure 1 , Figure 1 A schematic flow chart of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0025] Please refer to Figure 2 , Figure 2 A schematic diagram of the principle of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0026] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a specific process of a multi-viewpoint autonomous collaborative positioning method provided by the present invention.

[0027] The present invention provides a multi-viewpoint autonomous collaborative positioning method, comprising: 101: Determine multiple local maps and multiple keyframe information of the environment in which the multiple agents are located based on a preset simultaneous localization and mapping algorithm; the multiple agents are configured to obtain multi-viewpoint local environment information from different positions and perspectives using the onboard visual sensors; the multi-viewpoint local environment information corresponds to the multiple local maps and multiple keyframe information; 102: determining a constraint relationship between multiple local maps based on multiple keyframe information, and determining a local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes a pose constraint of adjacent keyframes; 103: Fusing multiple local maps according to the local map coordinate system transformation matrix to obtain a global map, and determining global positioning information according to the global map.

[0028] This embodiment uses a drone swarm as an example to illustrate the specific implementation of a multi-viewpoint autonomous collaborative localization method. Assume that the drone swarm consists of multiple unmanned aerial vehicles (UAVs), each equipped with a visual sensor (such as a camera) to acquire local information about the environment.

[0029] Each drone uses its onboard visual sensors to obtain local environmental information from different locations and perspectives. This information includes image data, feature points, and more. Based on a pre-defined simultaneous localization and mapping (SLAM) algorithm, each drone independently constructs a local map and extracts keyframe information. A keyframe is a representative frame selected from a continuous image sequence. For example, feature points can be extracted from a keyframe, and the pose change of each keyframe relative to the previous keyframe can be calculated. These pose changes and feature point information constitute the local map. Next, the constraints between the local maps of different drones need to be determined. This step is achieved by matching keyframes from different drones. Specifically, by comparing feature points in the keyframes, matching pairs of feature points are found. These matching pairs provide relative pose information between the local maps. Using these matching pairs of feature points, the coordinate transformation matrix between the local maps can be calculated using epipolar geometry. The local maps of different drones can then be fused based on the coordinate transformation matrix. Specifically, map points in one local map are projected into the coordinate system of another local map using the transformation matrix. In this way, multiple local maps can be fused into a single global map. This global map incorporates the local map information of all drones, providing a unified description of the environment for the entire drone swarm. Finally, based on this fused global map, the positioning of each drone within the global environment can be determined.

[0030] The multi-viewpoint autonomous collaborative positioning method of the present invention can effectively improve the positioning accuracy and collaborative ability of drone clusters in complex environments; give each drone greater autonomy, so that each drone can independently complete SLAM work during operation; it can reduce the system's dependence on a single drone, improve its adaptability to complex scenarios, and at the same time improve the system's compatibility and portability.

[0031] Cameras can be monocular, binocular, or RGB-D. The drone platform, which will carry the camera, places strict restrictions on size, weight, and power consumption. To reduce drone costs and enable a "swarm" operation mode, monocular cameras are the best choice.

[0032] The intelligent agent can also be a mobile platform such as a self-driving car, an automated guided vehicle (AGV), or an autonomous underwater vehicle (AUV).

[0033] For cameras, image acquisition is continuous, and appropriate image frames need to be selected as key frames for storage and subsequent calculations. This can reduce the requirements for the device while having little impact on the results, and use limited computing resources for more optimized operations. For example, the following steps are used to filter image frames and extract key frames. Step 1: After the last repositioning is completed, 20 frames of data have passed, or the number of key frames is less than 20, and the conditions are met to proceed to the next judgment. Step 2: After the last key frame is inserted, 20 frames have passed, and the local mapping module is in an idle state, and the conditions are met to proceed to the next judgment. Step 3: The current frame tracks more than 15 feature points, and the conditions are met to proceed to the next judgment. Step 4: The current number of tracking points is less than 90% of the reference key frame points, and the condition is met to determine it as a key frame.

[0034] Steps 1 and 2 directly affect the density of keyframes over time, and their parameters must take into account both computing resources and camera frame rate. Step 3 is designed to remove blurry frames and minimize the effects of camera shake. The reference keyframe in step 4 is the most recent keyframe before the current frame, ensuring a certain degree of distinction between keyframes. This prevents duplicate frames from being generated by the agent's dwell time in a specific area during operation. Duplicate frames can increase the singularity of optimization parameters and significantly reduce the effectiveness of the optimization.

[0035] As a preferred embodiment, the visual sensor is a camera, and the multi-viewpoint local environment information is a multi-viewpoint local environment image; based on a preset synchronous positioning and mapping algorithm, multiple local maps and multiple key frame information of the environment where multiple intelligent agents are located are determined, including: obtaining a multi-viewpoint local environment image collected by a camera; obtaining the camera pose information of the image key frame based on the multi-viewpoint local environment image and the camera's intrinsic parameter matrix; constructing multiple local maps of the environment where multiple intelligent agents are located based on the camera pose information of the image key frame and the multi-viewpoint local environment image.

[0036] In this embodiment, each drone uses its onboard camera to capture image information of the local environment from different positions and perspectives. This image information is the basis for subsequent positioning and mapping.

[0037] In a single drone, it includes a visual odometry module, a mapping module, and a communication module with the server.

[0038] The visual odometry (VO) module calculates not only the distance traveled but also the trajectory and pose of the camera. The VO module's inputs are multi-view local environment images captured by the camera and the camera's intrinsic parameter matrix, which is fixed and obtained through camera calibration. The VO module's output is the camera position and pose for each keyframe.

[0039] Mapping refers to the process of map construction, and the forms of maps are diverse and can be determined according to specific application scenarios. This embodiment uses a metric map, which accurately represents the positional relationship of objects in the map, and is divided into sparse maps, semi-dense maps, and dense maps according to the sparsity of map points. The mapping module constructs multiple local maps of the environment where multiple intelligent agents are located based on the camera pose information of the image key frames and the local environment images from multiple viewpoints. The local map on the drone side contains a camera pose map and map points. For parts of the image where the grayscale changes significantly, a sparse map is constructed, and the map points represent the feature points of a certain key frame. In order to reduce the storage and communication pressure of the drone, the size of the local map on the drone side is limited to N key frames, that is, the local map only contains the feature points in the latest N key frames. When each new key frame is obtained, it will be assigned a unique number consisting of the drone number and the corresponding timestamp, which is convenient for the server to store and manage.

[0040] The communication module sends processed image key frames, camera pose information, and local map point information to the server. The communication module is one-to-one with the agent, ensuring that each agent can work independently without interference.

[0041] The communication module in the CoSLAM (Collaborative Simultaneous Localization and Mapping) framework is a crucial component connecting the client and server. Its primary function is to transmit and process information between the two ends. On the server side, a communication processing interface, one-to-one with the drone, handles data reception and some information processing, such as loop detection. This process requires significant computing resources, making it difficult for non-PC drone computing devices to handle.

[0042] As a preferred embodiment, the constraint relationship also includes loop detection constraints; determining the constraint relationship between multiple local maps based on multiple key frame information, including: determining a pose graph based on the camera pose information of multiple image key frames; the nodes of the pose graph represent the camera poses of the key frames; the edges of the pose graph represent the relative pose constraints between the connected key frames; based on the camera pose information of multiple image key frames, a loop detection algorithm is used to use the constraints between the current frame and the historical frames as loop detection constraints.

[0043] In this embodiment, the server is a base station with strong computing power. Within the server, a data processing module corresponds to each drone and processes data originating from that specific drone. The server also includes a storage module containing a global map and keyframe library, storing important system data by time. To achieve matching between local maps and fusion of the global map, the server incorporates a location recognition module, a map fusion module, and an optimization module.

[0044] In actual system operation, any sensor acquisition process is inherently noisy. Furthermore, since image acquisition involves downsampling, quantizing a continuous scene into discrete pixel values, quantization error is not negligible. Therefore, a unified optimization requires integrating multiple frames. This optimization process requires the introduction of constraints. Constraints come from two sources: those introduced by matching feature points in adjacent frames and those introduced by the loop closure detection module. After constructing a loss function based on the corresponding optimization objectives (camera pose and map point coordinates), these constraints are used to optimize the system and obtain a globally consistent trajectory and map. The basic idea is to empower the agent to recognize previously visited scenes. During system operation, when the agent recognizes a location previously reached (at a time interval), a constraint is established between the current frame and the previous frame. Location recognition can be achieved using either classic image matching methods or machine learning-based approaches. The difference in approach primarily affects recognition accuracy and time efficiency.

[0045] The position recognition module obtains the constraint relationship between local maps by performing inter-frame feature matching on the camera pose information of image key frames collected from different drones.

[0046] Based on the keyframe representation of the map, a pose graph is constructed, closed-loop constraints are established, and general graph optimization (G2O) is used for overall optimization to maintain the overall consistency of the map. The pose graph is a graph structure that represents the poses of keyframes and the relative pose information between keyframes. In the pose graph, vertices represent the camera poses of keyframes, and edges represent the relative pose constraints between the two keyframes they connect.

[0047] Whenever a local map and keyframe information is received from a drone, it is stored by number. During the server storage process, the size of the local map is not limited by the number of keyframes, and all local map points from the initialization phase are recorded. Therefore, during the local map loop detection process, when a drone is detected to have arrived at a place it has been to before, even though this part of the map data may not exist on the drone side, the server can still optimize this. The server stores all received drone keyframe information, including the camera pose corresponding to the feature points and keyframes, as well as the BoW vector. A variable-length keyframe storage unit is maintained for each drone in the server. When a keyframe is received from the drone, the keyframe number and keyframe information are recorded at the same time. The keyframe number corresponds one-to-one with the keyframe to ensure that there is no confusion during the matching process of different local maps.

[0048] As the system runs, errors will continue to accumulate, including rotation errors and unique errors. This embodiment uses a loop detection method to reduce the accumulated errors. The function of the loop detection module is to add additional map constraints to provide necessary conditions for map optimization. By detecting that the camera takes the same position at different times, it is called a loop, and this is used to increase the constraints between key frames to facilitate better globally consistent optimization results. The loop detection constraints include constraints on the same drone key frame. Similarly, if loops are also detected between different local maps, new constraints can be added. At the same time, since loop detection provides constraints between the current frame and the historical frame, after the inter-frame tracking algorithm is lost due to the number of feature points or other external reasons, the inter-frame constraints can be used to reposition the camera. This method can reduce problems during system operation.

[0049] The server continuously checks for newly entered keyframes, regardless of whether a local map match has already been completed. This testing process involves two keyframe loopback detection: loopback detection of the same agent's keyframes to eliminate cumulative errors, and positional identification of keyframes from different agents to add new constraints and reduce random errors in a single match. These two detections form two closed loops within the system, ensuring long-term stable and efficient operation.

[0050] It should be noted that to detect common regions between keyframes, a bag-of-words (BoW) approach is used. The bag-of-words (BoW) model treats image features as words and compares the types and number of words appearing in different frames to determine whether the same object exists between two frames. All words form a dictionary, and word searches are performed within this dictionary. Once the visual dictionary is obtained, keyframe features are converted into BoW vectors through a tree search. BoW vectors record the word number and weight associated with each word. When the upper-layer server communication interface receives keyframe information from the agent client, it first performs a tree search within the dictionary to obtain the corresponding BoW vectors, which are then stored by number. This vector is then tested for similarity with candidate keyframes to identify keyframes whose similarity meets a threshold. This step performs a coarse image matching operation and only determines matching relationships between keyframes. Pose and coordinate point calculations are based on feature points. Therefore, fine matching is required using the feature point-based matching method described earlier to determine matching relationships between feature points and optimize the results. The advantage of the bag-of-words model is that it can calculate the corresponding BoW vector for each newly entered key frame and store it. It can be directly called in the subsequent matching process without repeated calculation, which can greatly reduce the amount of matching operations.

[0051] As a preferred embodiment, a binary selection algorithm is used to select key frames from frames that are a preset number of frames away from the current frame.

[0052] The detection of loop closure is essentially an image recognition problem, which is carried out in the image recognition module of the CoSLAM system. To achieve the matching of the current frame and the historical frame, the first step is to find the frame to be matched. In this embodiment, a half-selection algorithm is adopted, assuming that it is easier to find a matching frame in a key frame closer to the new key frame time. Since the number of frames selected is logarithmically related to the running time, it can still have a higher time efficiency after a long run, avoiding too many program loops. Similarly, in order to avoid selecting a key frame that is too close to the current frame, screening is started from the distance of the current 20 key frames.

[0053] As a preferred embodiment, determining the local map coordinate system transformation matrix according to the constraint relationship includes: using an extreme geometry algorithm to calculate the pose transformation relationship between image key frames according to the constraint relationship to obtain the local map coordinate system transformation matrix.

[0054] After obtaining the matching relationship between the current image frame and the historical image frame, that is, obtaining the constraint relationship between the local maps, in this embodiment, the current relative posture of the two intelligent bodies is solved by epipolar geometry operations to obtain the local map coordinate system transformation matrix. According to the local map coordinate system transformation matrix, one of the local maps can be projected into another local map to form a larger fusion map. Specifically, through location recognition, when one of the drones 1 recognizes a similar scene, the other drone 2 is not there. The connection directly established by epipolar geometry is only the connection between the current drone 1 and the drone 2 at a certain moment in the past. After a period of time, the drone 2 detects a different environmental scene again, so it is necessary to calculate the SLAM process of camera 2 during this period. Assume that at a certain moment t, drone 1 arrives at a scene A that drone 2 arrived at at time k before drone 2, and drone 2 arrives at B at time t. Let the relative (previous moment) posture of drone 2 at this moment be , the relative motion relationship set of UAV 2 from time k to time t is expressed as follows , Homogeneous representation is: , Can be ordered , The cumulative motion of drone 2 during this period can be expressed as the homogeneous transformation matrix operation , Will i The position of UAV 2 at this moment is described in the following form , Then the pose of UAV 2 in the local coordinates at time k can be expressed homogeneously as .

[0055] Among them, in a three-dimensional coordinate system, it is usually necessary to describe the transformation between two coordinate systems. Six degrees of freedom are required, of which three are the position transformations of the origin of the coordinate system, which can be expressed by vectors. The other three degrees of freedom are the rotations with the three coordinate axes as normals. In the rectangular coordinate system, they can be expressed by the rotation matrix express. is the attitude matrix of UAV 2, is the position vector of UAV 2, The accumulated time motion of drone 2.

[0056] The transformation matrix T has a rather special structure: the upper left corner is the rotation matrix, the right side surrounds the translation, the lower left corner is the 0 vector, and the lower right corner is 1. This kind of matrix is also called a special Euclidean group.

[0057] During this process, the local map constructed based on drone 2 is transferred from the coordinate system at time t to the coordinate system at time k, preparing for map fusion (the map fusion process is performed based on drone 1's local map). This process unifies the scene at the associated drones. It should be noted that the communication module has an independent thread and a corresponding information buffer to ensure the smooth operation of other modules. During each work cycle, the communication module on drone 2 packages the newly generated keyframe information with the local map points after the previous cycle and publishes this message. After receiving this message, the server communication interface stores the keyframe and map points separately and records the numbers. When new data is received from drone A, a judgment is made. If the local map represented by drone A has been matched with the global map, that is, the transformation matrix T between the two coordinate systems is known, the server directly transforms the new local map points into global map points using the transformation matrix T, completing map fusion. If drone A's local map has not been matched with the global map, that is, the transformation matrix T is unknown, a keyframe-based map matching process is required, primarily performed by the map recognition module.

[0058] The map fusion process can occur between any UAVs. In this embodiment, UAV 1 and UAV 2 are still used as analysis objects. The above process calculates the relative position relationship between the two UAVs and unifies the coordinate time of the two UAVs. The map fusion process occurs in the local coordinates of the map of UAV 1 (assuming that UAV 1 is the upper unit). The map fusion module uses the relative position relationship (R, t) to compare the converted local map built based on UAV 2 with the map. Figure 1 For fusion, the homogeneous expression of map feature point fusion is as follows: , in, is the coordinate in the coordinate system at time k, is the coordinate before fusion, Locally Figure 2 The number of feature points in the figure. The superscript 2 is the parameter corresponding to UAV 2.

[0059] After fusion, the local Figure 1 In the k-time coordinate system, transferring it to the local map coordinate system of UAV 1 only requires a homogeneous transformation , in, is the change in the position of UAV 1 at time k and the starting time, is the number of feature points in the fused map, and the superscript 1 indicates the parameter corresponding to UAV 1.

[0060] As a preferred embodiment, it also includes: in a general graph optimization framework, map points and camera poses are used as nodes, and reprojection errors are used as edges, and a global optimization algorithm is used to globally optimize the global map; the optimization goal of the global optimization algorithm is to minimize the reprojection error; the reprojection error is the difference between the projected position of the map point in the camera and the actual observed position.

[0061] Taking into account the actual situation when position recognition occurs, the epipolar geometry method is used to find the initial solution when calculating the relative pose of the two drones at time k. The local maps initially obtained by the drones need to be optimized because the system may have accumulated time errors and observation errors. In this embodiment, a global optimization (BA, Bundle Adjustment) algorithm is used to reduce errors, improve accuracy, and obtain the optimized solution (R, t). The global map can be understood as a fusion map composed of all local maps. Specifically, when taking a picture, the camera projects the spatial point onto the image, which is the first projection. These images are used for feature matching, and then the matched feature points are triangulated to obtain the position of the three-dimensional spatial point. Finally, the calculated coordinates of the three-dimensional point (with error) are used together with the camera intrinsic parameter matrix and the attitude transformation matrix of different key frames for secondary projection, that is, reprojection. The optimization goal adopted for optimizing the map points is to minimize the reprojection error. The optimization module constructs the optimization problem as shown below: , , in, c Number the drone. m i For map points, is the prediction point, is the Huber kernel function, which can effectively remove the influence of large errors. is the optimized parameter set, The variable value that makes the objective function reach the minimum value, For the i The coordinates of the 3D map points (their positions in 3D space), For intelligent agents c All optimization parameters of For intelligent agents c No. p The pose transformation matrix of the key frame, is the visibility indicator variable (Binary Indicator), x is the value of the reprojection error, that is, the Euclidean distance between the observation value and the projection value, bis the threshold parameter of the Huber kernel function, which controls the switching boundary between the quadratic term and the linear term. The optimization target is the three-dimensional space coordinate value and the state transition matrix. The entire optimization process can be placed in the g2o graph optimization framework, with the optimization amount as the node and the constraint relationship as the edge, which is easy to program. The constraint relationship here includes the direct pose constraints of adjacent key frames, as well as the constraints obtained by processes such as loop detection and key frame matching between local maps. Since the BA process has a large demand for computing resources in the entire system and takes a long time, it is only performed on the server. During the optimization process, the corresponding local map communication interface function is closed, and only one buffer is maintained to receive data to avoid map inconsistency problems during the process.

[0062] The multi-viewpoint autonomous collaborative positioning system provided by the present invention is described below. The multi-viewpoint autonomous collaborative positioning system described below and the multi-viewpoint autonomous collaborative positioning method described above can be referenced to each other.

[0063] Please refer to Figure 4 , Figure 4 This is a structural diagram of a multi-viewpoint autonomous collaborative positioning system provided by the present invention.

[0064] The present invention also provides a multi-viewpoint autonomous collaborative positioning system, including: a local map determination module 401, which is used to determine multiple local maps and multiple key frame information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; multiple intelligent agents are used to obtain local environment information of multiple viewpoints from different positions and perspectives through the carried visual sensors; the local environment information of multiple viewpoints corresponds to multiple local maps and multiple key frame information; a constraint relationship determination module 402, which is used to determine the constraint relationship between multiple local maps based on multiple key frame information, and determine the local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes the posture constraints of adjacent key frames; a global positioning module 403, which is used to fuse multiple local maps according to the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information based on the global map.

[0065] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5As shown, the electronic device may include: a processor 501, a communications interface 502, a memory 503, and a communications bus 504. The processor 501, the communications interface 502, and the memory 503 communicate with each other via the communications bus 504. The processor 501 may invoke logic instructions in the memory 503 to execute a multi-view autonomous collaborative localization method, which includes: determining multiple local maps and multiple keyframe information of the environment in which multiple intelligent agents are located based on a preset simultaneous localization and mapping algorithm; the multiple intelligent agents are configured to obtain multi-view local environment information from different positions and perspectives using onboard visual sensors; the multi-view local environment information corresponds to the multiple local maps and multiple keyframe information; determining constraint relationships between the multiple local maps based on the multiple keyframe information, and determining a local map coordinate system transformation matrix based on the constraint relationships; the constraint relationships include pose constraints between adjacent keyframes; and fusing the multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, thereby determining global positioning information based on the global map.

[0066] Furthermore, the logic instructions in the aforementioned memory 503 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-viewpoint autonomous collaborative positioning method provided by the above methods, which includes: determining multiple local maps and multiple key frame information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; multiple intelligent agents are used to obtain multi-viewpoint local environment information from different positions and perspectives through the onboard visual sensors; the multi-viewpoint local environment information corresponds to multiple local maps and multiple key frame information; determining the constraint relationship between multiple local maps based on multiple key frame information, and determining the local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes the posture constraints of adjacent key frames; and fusing multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information based on the global map.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-viewpoint autonomous collaborative positioning method provided by the above-mentioned methods, the method comprising: determining multiple local maps and multiple key frame information of the environment in which multiple intelligent agents are located based on a preset synchronous positioning and mapping algorithm; multiple intelligent agents are used to obtain local environmental information of multiple viewpoints from different positions and perspectives through the visual sensors they carry; the local environmental information of multiple viewpoints corresponds to multiple local maps and multiple key frame information; determining the constraint relationship between multiple local maps based on multiple key frame information, and determining the local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes the posture constraints of adjacent key frames; and fusing multiple local maps based on the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information based on the global map.

[0069] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0070] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-viewpoint autonomous collaborative positioning method, characterized in that: include: Determine multiple local maps and multiple keyframe information of the environment where multiple agents are located based on a preset simultaneous localization and mapping algorithm; The plurality of intelligent agents are used to obtain multi-viewpoint local environment information from different positions and perspectives through the visual sensors carried therein; the multi-viewpoint local environment information corresponds to the plurality of local maps and the plurality of key frame information; Determining a constraint relationship between the plurality of local maps based on the plurality of key frame information, and determining a local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes a pose constraint of adjacent key frames; The multiple local maps are fused according to the local map coordinate system transformation matrix to obtain a global map, so as to determine global positioning information according to the global map.

2. The multi-viewpoint autonomous collaborative positioning method according to claim 1, characterized in that: The visual sensor is a camera, and the multi-viewpoint local environment information is a multi-viewpoint local environment image; The method of determining multiple local maps and multiple key frame information of the environment in which multiple agents are located based on a preset synchronous positioning and mapping algorithm includes: Acquire the local environment images of the multiple viewpoints captured by the camera; Obtaining camera pose information of an image key frame according to the multi-view local environment image and the intrinsic parameter matrix of the camera; Based on the camera pose information of the image key frame and the multi-viewpoint local environment image, multiple local maps of the environment where the multiple intelligent agents are located are constructed.

3. The multi-viewpoint autonomous collaborative positioning method according to claim 2, characterized in that: The constraint relationship also includes loop detection constraints; The determining of the constraint relationship between the plurality of local maps according to the plurality of key frame information includes: Determining a pose graph based on the camera pose information of the plurality of image key frames; the nodes of the pose graph represent the camera poses of the key frames; the edges of the pose graph represent the relative pose constraints between the connected key frames; According to the camera pose information of the plurality of image key frames, a loop detection algorithm is adopted to set the constraint between the current frame and the historical frames as the loop detection constraint.

4. The multi-viewpoint autonomous collaborative positioning method according to claim 1, characterized in that: Use a binary selection algorithm to filter key frames from frames that are a preset number of frames away from the current frame.

5. The multi-viewpoint autonomous collaborative positioning method according to claim 3, characterized in that: Determining the local map coordinate system transformation matrix according to the constraint relationship includes: According to the constraint relationship, an epipolar geometry algorithm is used to calculate the pose transformation relationship between the image key frames to obtain the local map coordinate system transformation matrix.

6. The multi-viewpoint autonomous collaborative positioning method according to any one of claims 1 to 5, characterized in that: Also includes: In a general graph optimization framework, map points and camera poses are used as nodes, and reprojection errors are used as edges. A global optimization algorithm is used to globally optimize the global map. The optimization goal of the global optimization algorithm is to minimize the reprojection error; the reprojection error is the difference between the projected position of the map point in the camera and the actual observed position.

7. A multi-viewpoint autonomous collaborative positioning system, characterized in that: include: A local map determination module is used to determine multiple local maps and multiple keyframe information of the environment where multiple agents are located based on a preset simultaneous positioning and mapping algorithm; The plurality of intelligent agents are used to obtain multi-viewpoint local environment information from different positions and perspectives through the visual sensors carried therein; the multi-viewpoint local environment information corresponds to the plurality of local maps and the plurality of key frame information; A constraint relationship determination module, configured to determine the constraint relationship between the plurality of local maps based on the plurality of key frame information, and determine the local map coordinate system transformation matrix based on the constraint relationship; the constraint relationship includes the position and posture constraints of adjacent key frames; The global positioning module is used to fuse the multiple local maps according to the local map coordinate system transformation matrix to obtain a global map, so as to determine the global positioning information according to the global map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the multi-viewpoint autonomous collaborative positioning method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-viewpoint autonomous collaborative positioning method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-viewpoint autonomous collaborative positioning method according to any one of claims 1 to 6 is implemented.

Images