A relocation method, device, electronic device and storage medium

Abstract

The present application relates to a method, device, electronic device and storage medium for repositioning, and to the field of computer technology. The method comprises: semantically matching a multi-camera image acquired based on an initial anchor point with a multi-camera image historically anchored to obtain an estimated position of the current anchor point in a known map, and then determining a local historical anchor point frame in the known map, the local historical anchor point frame being a historical anchor point frame with the closest Euclidean distance to the estimated position, and then performing multi-level matching and least squares fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the pose of the initial anchor point, and then determining the pose of the current anchor point based on the pose of the initial anchor point. The present application calculates the initial pose of the repositioning based on the prior information, and can improve the accuracy of the initial pose obtained during repositioning even in the case of relatively large noise.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a relocation method, device, electronic device and storage medium. Background Art

[0002] In the field of autonomous driving or automated robotics, a smart terminal is usually equipped with multiple cameras to perceive the environment around the smart terminal and interact with the surrounding environment. Specifically, the surrounding environment is modeled by multiple cameras to assist the smart terminal in obtaining its relative position in the environment.

[0003] The related technology involves a multi-camera based positioning and mapping technology to obtain the initial posture during repositioning. Specifically, in the initialization stage, it is assumed that there is a certain overlapping area of ​​viewing angles between cameras. During the mapping process, the posture of the camera is tracked by assuming uniform motion, and the initial posture during repositioning is calculated by combining the generalized perspective-three-point (GP3P) algorithm with the uncalibrated perspective-n-point (UPNP) algorithm.

[0004] During the research process, the inventors found that when calculating the initial pose during repositioning through the GP3P algorithm and the UPNP algorithm, when the time span between positioning and mapping is relatively large, it is usually necessary to perform a large number of samplings in the matching of feature points with more error information and map points through random sampling consistency, so as to obtain the pose of the anchor points of multiple cameras through linear matrix decomposition. In the absence of prior information, it is often impossible to verify the correctness of the linear matrix decomposition solution when the noise is relatively large. That is, when the noise is relatively large, the accuracy of the initial pose during repositioning calculated by the above method is low. Summary of the invention

[0005] The purpose of this application is to provide a relocation method, device, electronic device and storage medium to solve at least one of the above technical problems.

[0006] The above invention objectives of the present application are achieved through the following technical solutions:

[0007] In a first aspect, a relocation method is provided, comprising:

[0008] Semantically match the multi-camera images acquired based on the initial anchor point with the multi-camera images of the historical anchor point to obtain the estimated position of the current anchor point in the known map;

[0009] Determining a local historical anchor point frame in the known map, the local historical anchor point frame being a historical anchor point frame having a closest Euclidean distance to the estimated position;

[0010] Perform multi-level matching and least squares fitting on the feature points saved in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and pose of the initial anchor point;

[0011] The pose of the current anchor point is determined based on the pose of the initial anchor point.

[0012] In a possible implementation, the multi-camera image acquired based on the initial anchor point stores a feature descriptor corresponding to the initial anchor point, and the multi-camera image historically anchored stores a historical feature descriptor; and the multi-camera image acquired based on the initial anchor point is semantically matched with the multi-camera image historically anchored to obtain an estimated position of the current anchor point in the known map, including:

[0013] Determine the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point;

[0014] Determine the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor;

[0015] Calculate the vector distances between the semantic BOW of the current anchor point and the semantic BOWs corresponding to each of the historical anchor point frames;

[0016] Determine the historical anchor point frame with the smallest vector distance;

[0017] The position of the historical anchor point frame with the smallest vector in the known map is determined as the estimated position of the current anchor point in the known map.

[0018] In another possible implementation, determining a local historical anchor point frame in the map includes:

[0019] Based on the estimated position of the current anchor point in the known map and the positions of each historical anchor point frame in the known map, determine each Euclidean distance, wherein each Euclidean distance is the Euclidean distance between the historical anchor point frame with the smallest vector distance and each historical anchor point frame;

[0020] A historical anchor point frame with the smallest Euclidean distance is determined from the various historical anchor point frames as the local anchor point frame.

[0021] In another possible implementation, performing multi-level matching and least squares fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point includes:

[0022] Extracting the map points corresponding to each camera stored in the anchor point frame with the smallest Euclidean distance and the historical anchor point frame to obtain a map point set;

[0023] determining at least one map point from the set of map points;

[0024] Projecting the at least one map point onto a corresponding camera plane respectively;

[0025] Extracting a set of feature points satisfying a first preset condition from the current image based on the projection information;

[0026] Based on the feature point set that meets the first preset condition and the map point set, determining first matching pair sets corresponding to each camera, wherein the first matching pair set corresponding to any camera includes: a plurality of first matching pairs, wherein the first matching pairs are map points and feature points that match the map points, and the vector distance between the feature descriptor of the map points and the feature descriptor of the feature points that match the map points is the smallest;

[0027] Determining an initial pose of the initial anchor point based on the first set of matching pairs;

[0028] Projecting the at least one map point to a camera plane that satisfies a second preset condition, wherein the camera plane that satisfies the second preset condition is a camera plane that is closest to the camera corresponding to each map point in a clockwise Euclidean distance;

[0029] Extracting a set of feature points satisfying a third preset condition from the current image based on the projection information;

[0030] Based on the feature point set that meets the third preset condition and the map point set, determining a second matching pair set, where the second matching pair set is a matching pair set corresponding to the camera that meets the second preset condition;

[0031] Based on the first matching pair set and the second matching pair set respectively corresponding to each camera, the initial pose of the initial anchor point is optimized to obtain the pose of the initial anchor point.

[0032] In another possible implementation, determining the position and posture of the current anchor point based on the position and posture of the initial anchor point includes:

[0033] Get the rigid body connection information between the current anchor point and multiple cameras and the texture information of the multiple camera images corresponding to the current anchor point;

[0034] Based on the rigid body connection information between the current anchor point and multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point, each third matching pair set is determined, and any third matching pair set includes: a matching set between a first feature point corresponding to any camera connected to the current anchor point and a map point of a known map;

[0035] Recursively extrapolate the matching information of the previous anchor frame to the current anchor frame to obtain the predicted matching information of the second feature point in the current anchor frame;

[0036] Based on the predicted matching information of the second feature point in the current anchor point frame, determining a plurality of fourth matching pair sets, wherein any fourth matching pair set includes: a matching set between the second feature point corresponding to any camera and a map point of a known map;

[0037] Merge and remove duplicates of the third matching pair sets and the fourth matching pair sets corresponding to each other to obtain matching information of the current anchor point frame;

[0038] Based on the matching information of the current anchor point frame, a fitting process is performed by a least square method based on the anchor point to obtain the position and posture of the current anchor point.

[0039] In another possible implementation, the determining of a plurality of third matching pair sets based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point includes:

[0040] Get a specific pose from a preset sensor;

[0041] Based on the specific pose and the pose of the initial anchor point, determining an estimated pose corresponding to the anchor point frame at the current moment;

[0042] Based on the estimated position and posture corresponding to the anchor frame at the current moment, determining a reference anchor frame from a known map, wherein the reference anchor frame is a reference anchor frame corresponding to the anchor frame at the current moment;

[0043] Projecting the map points corresponding to each camera stored in the reference anchor frame to the camera image corresponding to each anchor frame at the current moment, to obtain a feature point set corresponding to the anchor frame at the current moment, wherein the feature point set is a feature point set corresponding to each camera;

[0044] The third matching pair sets are determined based on the feature descriptors stored in the known map and the feature descriptors corresponding to the feature points in the feature point set.

[0045] In another possible implementation, the matching information of the anchor frame at the previous moment includes: each camera information corresponding to the anchor frame at the previous moment, each camera information including a matching set of corresponding feature points and map points;

[0046] The step of recursively extending the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame includes:

[0047] Based on each matching set, the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is predicted, and the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is the position of each feature point at the current moment in the camera image corresponding to the current moment.

[0048] In another possible implementation, the determining a plurality of fourth matching pair sets based on the predicted matching information of some feature points in the current anchor point frame includes:

[0049] Based on the positions of the feature points of the image at the previous moment in the camera image corresponding to the current moment, the matching points corresponding to the feature points at the current moment are determined;

[0050] constructing a plurality of fifth matching pair sets based on each feature point at the current moment and the matching points respectively corresponding to each feature point at the current moment;

[0051] The respective corresponding fourth matching pair sets are determined based on the plurality of fifth matching pair sets.

[0052] In another possible implementation, the matching information of the current anchor point frame is matched based on the matching information of the current anchor point frame, and the pose of the current anchor point is obtained by fitting the matching information based on the anchor point using a least square method. The method includes:

[0053] Based on the matching information of the current anchor point frame, the estimated position and posture corresponding to the anchor point frame at the current moment is optimized by using the anchor point-based least squares method to obtain the position and posture of the current anchor point.

[0054] In another possible implementation, the determining the position and posture of the current anchor point based on the position and posture of the initial anchor point further includes:

[0055] Determine matching information between the current anchor point frame and the local map based on a multi-combination matching method between the current anchor point frame and the local map;

[0056] Based on the matching information between the current anchor point frame and the local map, the position and posture of the initial anchor point are optimized to obtain the optimized position and posture of the current anchor point.

[0057] In another possible implementation, the position and posture of the current anchor point includes: the position of the anchor point frame at the current moment in the known map;

[0058] The method of determining the matching information between the current anchor point frame and the local map based on the multi-combination matching between the current anchor point frame and the local map also includes:

[0059] Extracting a local map from the known map based on the position of the anchor frame at the current moment in the map;

[0060] The method of determining the matching information between the current anchor point frame and the local map based on the multi-combination matching between the current anchor point frame and the local map includes:

[0061] Projecting the map points corresponding to the cameras stored in the anchor point frame of the local map to the cameras corresponding to the current anchor point frame;

[0062] Based on the projection information, determining a plurality of sixth matching pair sets;

[0063] Projecting the map points corresponding to each camera stored in the anchor point frame of the local map to the first preset camera corresponding to the current anchor point frame, wherein the first preset camera corresponding to the current anchor point frame is the camera with the closest Euclidean distance to each camera;

[0064] Based on the projection information, determining a plurality of seventh matching pair sets;

[0065] Projecting the map point corresponding to each camera stored in the anchor point frame of the local map to the second preset camera corresponding to the current anchor point frame, wherein the second preset camera corresponding to the current anchor point frame is the camera with the farthest Euclidean distance from each camera;

[0066] Based on the projection information, a plurality of eighth matching pair sets are determined.

[0067] In another possible implementation, the optimizing the position and posture of the initial anchor point based on the matching information between the current anchor point frame and the local map to obtain the optimized position and posture of the current anchor point includes:

[0068] Based on the multiple sixth matching pair sets, the multiple seventh matching pair sets, and the multiple eighth matching pair sets, the pose of the initial anchor point is optimized to obtain an optimized pose of the current anchor point.

[0069] In a second aspect, a relocation device is provided, comprising:

[0070] The semantic matching module is used to semantically match the multi-camera images acquired based on the initial anchor point with the multi-camera images of the historical anchor point to obtain the estimated position of the current anchor point in the known map;

[0071] A first determination module is used to determine a local historical anchor point frame in the known map, wherein the local historical anchor point frame is a historical anchor point frame with the shortest Euclidean distance to the estimated position;

[0072] The matching and fitting module is used to perform multi-level matching and least squares fitting on the feature points saved in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point;

[0073] The second determination module is used to determine the position and posture of the current anchor point based on the position and posture of the initial anchor point.

[0074] In a possible implementation, the multi-camera image acquired based on the initial anchor point stores a feature descriptor corresponding to the initial anchor point, and the multi-camera image historically anchored stores a historical feature descriptor; when the semantic matching module semantically matches the multi-camera image acquired based on the initial anchor point with the multi-camera image historically anchored to obtain an estimated position of the current anchor point in a known map, it is specifically used to:

[0075] Determine the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point;

[0076] Determine the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor;

[0077] Calculate the vector distances between the semantic BOW of the current anchor point and the semantic BOWs corresponding to each of the historical anchor point frames;

[0078] Determine the historical anchor point frame with the smallest vector distance;

[0079] The position of the historical anchor point frame with the smallest vector in the known map is determined as the estimated position of the current anchor point in the known map.

[0080] In another possible implementation, when the first determination module determines the local historical anchor point frame in the map, it is specifically used to:

[0081] Based on the estimated position of the current anchor point in the known map and the positions of each historical anchor point frame in the known map, determine each Euclidean distance, wherein each Euclidean distance is the Euclidean distance between the historical anchor point frame with the smallest vector distance and each historical anchor point frame;

[0082] A historical anchor point frame with the smallest Euclidean distance is determined from the various historical anchor point frames as the local anchor point frame.

[0083] In another possible implementation, when the matching and fitting module performs multi-level matching and least squares fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point, it is specifically used to:

[0084] Extracting the map points corresponding to each camera stored in the anchor point frame with the smallest Euclidean distance and the historical anchor point frame to obtain a map point set;

[0085] determining at least one map point from the set of map points;

[0086] Projecting the at least one map point onto a corresponding camera plane respectively;

[0087] Extracting a set of feature points satisfying a first preset condition from the current image based on the projection information;

[0088] Based on the feature point set that meets the first preset condition and the map point set, determining first matching pair sets corresponding to each camera, wherein the first matching pair set corresponding to any camera includes: a plurality of first matching pairs, wherein the first matching pairs are map points and feature points that match the map points, and the vector distance between the feature descriptor of the map points and the feature descriptor of the feature points that match the map points is the smallest;

[0089] Determining an initial pose of the initial anchor point based on the first set of matching pairs;

[0090] Projecting the at least one map point to a camera plane that satisfies a second preset condition, wherein the camera plane that satisfies the second preset condition is a camera plane that is closest to the camera corresponding to each map point in a clockwise Euclidean distance;

[0091] Extracting a set of feature points satisfying a third preset condition from the current image based on the projection information;

[0092] Based on the feature point set that meets the third preset condition and the map point set, determining a second matching pair set, where the second matching pair set is a matching pair set corresponding to the camera that meets the second preset condition;

[0093] Based on the first matching pair set and the second matching pair set respectively corresponding to each camera, the initial pose of the initial anchor point is optimized to obtain the pose of the initial anchor point.

[0094] In another possible implementation, when the second determination module determines the position and posture of the current anchor point based on the position and posture of the initial anchor point, it is specifically configured to:

[0095] Get the rigid body connection information between the current anchor point and multiple cameras and the texture information of the multiple camera images corresponding to the current anchor point;

[0096] Based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point, each third matching pair set is determined, and any third matching pair set includes: a matching set between a first feature point corresponding to any camera connected to the current anchor point and a map point of the map;

[0097] Recursively push the matching information of the previous anchor frame to the current anchor frame to obtain the predicted matching information of some feature points in the current anchor frame;

[0098] Based on the predicted matching information of the second feature point in the current anchor point frame, determining a plurality of fourth matching pair sets, wherein any fourth matching pair set includes: a matching set between the second feature point corresponding to any camera and a map point of a known map;

[0099] Merge and remove duplicates of the third matching pair sets and the fourth matching pair sets corresponding to each other to obtain matching information of the current anchor point frame;

[0100] Based on the matching information of the current anchor point frame, a fitting process is performed by a least square method based on the anchor point to obtain the position and posture of the current anchor point.

[0101] In another possible implementation, when the second determination module determines the plurality of third matching pair matching sets based on the rigid connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point, the second determination module is specifically configured to:

[0102] Get a specific pose from a preset sensor;

[0103] Based on the specific pose and the pose of the initial anchor point, determining an estimated pose corresponding to the anchor point frame at the current moment;

[0104] Based on the estimated position and posture corresponding to the anchor frame at the current moment, determining a reference anchor frame from a known map, wherein the reference anchor frame is a reference anchor frame corresponding to the anchor frame at the current moment;

[0105] Projecting the map points corresponding to each corresponding camera stored in the reference anchor frame to the camera image corresponding to the anchor frame at the current moment, to obtain a feature point set corresponding to the anchor frame at the current moment, wherein the feature point set is a feature point set corresponding to each camera;

[0106] The third matching pair sets are determined based on the feature descriptors stored in the known map and the feature descriptors corresponding to the feature points in the feature point set.

[0107] In another possible implementation, the matching information of the anchor frame at the previous moment includes: each camera information corresponding to the anchor frame at the previous moment, each camera information including a matching set of corresponding feature points and map points;

[0108] When the second determination module recursively extrapolates the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame, it is specifically used to:

[0109] Based on each matching set, the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is predicted, and the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is the position of each feature point at the current moment in the camera image corresponding to the current moment.

[0110] In another possible implementation, when the second determination module determines the plurality of fourth matching pair sets based on the predicted matching information of some feature points in the current anchor point frame, it is specifically configured to:

[0111] Based on the positions of the feature points of the image at the previous moment in the camera image corresponding to the current moment, the matching points corresponding to the feature points at the current moment are determined;

[0112] constructing a plurality of fifth matching pair sets based on each feature point at the current moment and the matching points respectively corresponding to each feature point at the current moment;

[0113] The respective corresponding fourth matching pair sets are determined based on the plurality of fifth matching pair sets.

[0114] In another possible implementation, when the second determination module performs fitting processing based on the matching information of the current anchor point frame and using the anchor point-based least squares method to obtain the pose of the current anchor point, it is specifically used to:

[0115] Based on the matching information of the current anchor point frame, the estimated position and posture corresponding to the anchor point frame at the current moment is optimized by using the anchor point-based least squares method to obtain the position and posture of the current anchor point.

[0116] In another possible implementation, the device further includes: a third determination module and an optimization module, wherein:

[0117] The third determination module is used to determine the matching information between the current anchor point frame and the local map based on a multi-combination matching method between the current anchor point frame and the local map;

[0118] The optimization module is used to optimize the position and posture of the initial anchor point based on the matching information between the current anchor point frame and the local map to obtain the optimized position and posture of the current anchor point.

[0119] In another possible implementation, the position and posture of the current anchor point includes: the position of the anchor point frame at the current moment in the known map;

[0120] The device further comprises: an extraction module, wherein:

[0121] The extraction module is used to extract a local map from the known map based on the position of the anchor point frame at the current moment in the map;

[0122] Wherein, when determining the matching information between the current anchor point frame and the local map based on the multi-combination matching method between the current anchor point frame and the local map, the third determination module is specifically used to:

[0123] Projecting the map points corresponding to the cameras stored in the anchor point frame of the local map to the cameras corresponding to the current anchor point frame;

[0124] Based on the projection information, determining a plurality of sixth matching pair sets;

[0125] Projecting the map points corresponding to each camera stored in the anchor point frame of the local map to the first preset camera corresponding to the current anchor point frame, wherein the first preset camera corresponding to the current anchor point frame is the camera with the closest Euclidean distance to each camera;

[0126] Based on the projection information, determining a plurality of seventh matching pair sets;

[0127] Projecting the map point corresponding to each camera stored in the anchor point frame of the local map to the second preset camera corresponding to the current anchor point frame, wherein the second preset camera corresponding to the current anchor point frame is the camera with the farthest Euclidean distance from each camera;

[0128] Based on the projection information, a plurality of eighth matching pair sets are determined.

[0129] In another possible implementation, when the optimization module optimizes the position and posture of the initial anchor point based on the matching information between the current anchor point frame and the local map to obtain the optimized position and posture of the current anchor point, it is specifically used to:

[0130] Based on the multiple sixth matching pair sets, the multiple seventh matching pair sets, and the multiple eighth matching pair sets, the pose of the initial anchor point is optimized to obtain an optimized pose of the current anchor point.

[0131] In a third aspect, an electronic device is provided, the electronic device comprising:

[0132] one or more processors;

[0133] Memory;

[0134] One or more applications, wherein one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: perform operations corresponding to the relocation method shown in any possible implementation of the first aspect.

[0135] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement a relocation method as shown in any possible implementation method of the first aspect.

[0136] In summary, the present application includes at least one of the following beneficial technical effects:

[0137] The present application provides a repositioning method, device, electronic device and storage medium. Compared with the related art in which the initial pose during repositioning is calculated by combining the GP3P algorithm with the UPNP algorithm, in the present application, semantic matching is performed between the multi-camera image obtained by the initial anchor point and the multi-camera image of the historical anchor to obtain the estimated position of the current anchor point in the known map, and the local historical anchor point frame can be determined in the known map according to the estimated position, and then the feature points stored in the local historical anchor point frame are respectively matched with the feature points extracted from each camera image by multi-level matching and least squares fitting to obtain the pose of the initial anchor point, so as to further determine the pose of the current anchor point according to the pose of the initial anchor point. That is, in the present application, the pose of the initial anchor point is obtained by the above calculation as prior information, and then the pose of the current anchor point (initial pose during repositioning) is calculated based on the prior information, so that even in the case of relatively large noise, the accuracy of the obtained initial pose during repositioning can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0138] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0139] Figure 1 It is a schematic diagram of a relocation method flow provided by an embodiment of the present application;

[0140] Figure 2 is a schematic diagram of a relocation device structure provided in an embodiment of the present application;

[0141] Figure 3 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0142] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0143] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0144] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. The term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " herein, unless otherwise specified, generally indicates that the objects associated before and after are in an "or" relationship.

[0145] The embodiment of the present application provides a repositioning method, which is applied to a repositioning system. In the system, an anchor point coordinate system relative to each camera is set. The anchor point coordinate system can theoretically be any coordinate system in a known three-dimensional space. The external parameter matrix of each camera relative to the anchor point coordinate system can be obtained through a known multi-camera calibration algorithm, which is marked as (where e stands for extrinsic, Indicates the camera's tag number. ). And through the known camera intrinsic calibration algorithm and the known camera calibration model, such as the pinhole camera model, the camera projection parameters are obtained. , the camera projection parameters can transform a point P in a three-dimensional space observed by the camera into an image pixel point p, using the parameters Represents the inverse projection parameters of the camera. The inverse projection parameters can convert the pixel coordinate point p on the image into the unit vector of the incident light corresponding to the point p, that is, .

[0146] The map information in the known map is mainly composed of multiple anchor frames, each anchor frame stores all the information of all cameras at the same time. The known map can be constructed through known algorithms including but not limited to the MCPTAM (Multi-Camera Parallel Tracking and Mapping, MCPTAM) algorithm or MultiCol SLAM algorithm, etc. In the embodiment of the present application, MultiCol Simultaneous Localization And Mapping (MultiCol SLAM) is taken as an example. The anchor frame stores the known parameters such as the internal and external parameters of each camera mentioned above, the map points observed by each camera, including the texture information and position information of each point in the map point, etc., as well as the semantic information output by each camera at each moment through the Distributed Bag of Words (DBOW) algorithm.

[0147] Furthermore, the present application provides a relocation method, such as Figure 1 As shown, the relocation method can be performed by an electronic device, which can be a server or a terminal device, or an unmanned vehicle or an unmanned robot, etc., wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application. The method may include:

[0148] Step S101: semantically match the multi-camera image acquired based on the initial anchor point with the multi-camera image anchored historically to obtain an estimated position of the current anchor point in a known map.

[0149] For the embodiment of the present application, before step S101, it may include: obtaining each camera image corresponding to the initial anchor point, and obtaining each camera image of the historical anchor. In the embodiment of the present application, the step of obtaining each image corresponding to the initial anchor point may be performed before the step of obtaining each camera image of the historical anchor, and may also be performed after the step of obtaining each camera image of the historical anchor, and may also be performed simultaneously with the step of obtaining each camera image of the historical anchor, which is not limited in the embodiment of the present application.

[0150] Furthermore, after obtaining the camera images corresponding to the initial anchor point and the camera images of the historical anchor points, semantic matching is performed between the camera images obtained based on the initial anchor point and the corresponding camera images of the historical anchors to identify the possible positions of the current anchor point in the known map, that is, the estimated position of the current anchor point in the known map.

[0151] Step S102: Determine a local historical anchor frame in a known map.

[0152] Among them, the local historical anchor point frame is the historical anchor point frame with the closest Euclidean distance to the estimated position.

[0153] Specifically, after obtaining the estimated position of the current anchor point in the known map through the above embodiment, the Euclidean distance between the estimated position and each historical anchor point frame is determined, and the historical anchor point frame with the closest Euclidean distance is determined as the local historical anchor point frame. In the embodiment of the present application, the local historical anchor point frame can be at least one frame.

[0154] Step S103: Perform multi-level matching and least squares fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point.

[0155] Step S104: determine the position and posture of the current anchor point based on the position and posture of the initial anchor point.

[0156] For the embodiment of the present application, after the pose of the initial anchor point is obtained through the above embodiment, the pose of the current anchor point is determined based on the pose of the initial anchor point. In the embodiment of the present application, the manner of determining the pose of the current anchor point based on the pose of the initial anchor point is not limited.

[0157] An embodiment of the present application provides a method for repositioning. Compared with the related art in which the initial pose during repositioning is calculated by combining the GP3P algorithm with the UPNP algorithm, in the embodiment of the present application, semantic matching is performed between the multi-camera image obtained by the initial anchor point and the multi-camera image of the historical anchor to obtain an estimated position of the current anchor point in a known map, and a local historical anchor point frame can be determined in the known map according to the estimated position, and then the feature points stored in the local historical anchor point frame are respectively matched with the feature points extracted from each camera image by multi-level matching and least squares fitting to obtain the pose of the initial anchor point, so as to further determine the pose of the current anchor point according to the pose of the initial anchor point. That is, in the present application, the pose of the initial anchor point is obtained by the above calculation as prior information, and then the pose of the current anchor point (initial pose during repositioning) is calculated based on the prior information, so that even in the case of relatively large noise, the accuracy of the obtained initial pose during repositioning can be improved.

[0158] Specifically, in the embodiment of the present application, the multi-camera image acquired based on the initial anchor point stores the feature descriptor corresponding to the initial anchor point, and the historically anchored multi-camera image stores the historical feature descriptor; on this basis, in step S101, the multi-camera image acquired based on the initial anchor point is semantically matched with the historically anchored multi-camera image to obtain the estimated position of the current anchor point in the known map, which may specifically include: step S1011 (not shown in the figure), step S1012 (not shown in the figure), step S1013 (not shown in the figure), step S1014 (not shown in the figure) and step S1015 (not shown in the figure), wherein,

[0159] Step S1011: determining the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point, and determining the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor.

[0160] For the embodiments of the present application, the step of determining the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point can be performed before determining the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor, or after determining the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor, or can be performed simultaneously with determining the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor, which is not limited in the embodiments of the present application.

[0161] For the embodiment of the present application, after obtaining each camera image rigidly connected to the initial anchor point, a common feature point list and feature descriptor are extracted for each camera image, and the extraction algorithm includes but is not limited to an Oriented Fast and Rotated Brief (ORB) algorithm, a Scale-invariant feature transform (SIFT) algorithm, a Speed-up robust features (SURF) algorithm, and the like.

[0162] Step S1013: Calculate the vector distances between the semantic BOW of the current anchor point and the semantic BOWs corresponding to each historical anchor point frame.

[0163] Furthermore, after obtaining the feature descriptors corresponding to x images, The feature descriptors corresponding to the images are merged into a set and passed through the DBOW algorithm to obtain the semantic BOW label of the initial anchor point: , then The BOW marker in the anchor frame stored in the known map is (j is the sequence number of the anchor frame in the known map), calculate the BOW vector distance in sequence, and obtain the distance of each vector.

[0164] Step S1014: determine the historical anchor point frame with the smallest vector distance.

[0165] For the embodiment of the present application, after obtaining the vector distances, the historical anchor frame with the smallest vector distance is determined. For example, the anchor frame with the smallest semantic BOW vector distance is obtained as anchor frame F.

[0166] Step S1015: determine the position of the historical anchor point frame with the smallest vector in the known map as the estimated position of the current anchor point in the known map.

[0167] For the embodiment of the present application, after obtaining the anchor frame F, the position of the anchor frame F stored in the known map is obtained. ,in, It is the estimated position of the current anchor frame in the known map.

[0168] Further, after obtaining the estimated position of the current anchor point in the known map, the local historical anchor point frame is determined in the map in step S102, which may specifically include: step S1021 (not shown in the figure) and step S1022 (not shown in the figure), wherein:

[0169] Step S1021: Determine each Euclidean distance based on the estimated position of the current anchor point in the known map and the positions of each historical anchor point frame in the known map.

[0170] Among them, each Euclidean distance is the Euclidean distance corresponding to the historical anchor point frame with the smallest vector distance and each historical anchor point frame.

[0171] Specifically, the position of the anchor frame F stored in the known map is , respectively, with the positions of the anchor frames stored in the known map (j is the serial number of the anchor frame in the known map) and each Euclidean distance is calculated using formula (1).

[0172] Wherein, formula (1) is:

[0173] Formula (1);

[0174] Among them, t F represents the estimated position of the current anchor frame (anchor frame F) in the known map, t j represents the position of anchor frame j in the known map, and d represents t F With t j The Euclidean distance between .

[0175] Step S1022: Determine the historical anchor point frame with the smallest Euclidean distance from each historical anchor point frame as the local anchor point frame.

[0176] For the embodiment of the present application, the Euclidean distance between each historical anchor frame and the anchor frame F is obtained by the above formula (1). After the Euclidean distance between each anchor frame and the anchor frame F is obtained, the m anchor frames whose Euclidean distance to the anchor frame F is closest are determined, and the m anchor frames whose Euclidean distance to the anchor frame F is closest are determined as local anchor frames.

[0177] It should be noted that m can be 1 or greater than 1, which is not limited in the embodiments of the present application.

[0178] Specifically, in step S103, multi-level matching and least squares fitting are performed on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point, which may specifically include: step S103a (not shown in the figure), step S103b (not shown in the figure), step S103c (not shown in the figure), step S103d (not shown in the figure), step S103e (not shown in the figure), step S103f (not shown in the figure), step S103g (not shown in the figure), step S103h (not shown in the figure), step S103i (not shown in the figure) and step Sj (not shown in the figure), wherein,

[0179] Step S103a: extract the map points corresponding to each camera stored in the anchor point frame with the smallest Euclidean distance and the historical anchor point frame to obtain a map point set.

[0180] Specifically, after obtaining the m initial anchor frames closest to the anchor frame F, the map points corresponding to each camera stored in the anchor frame F and the m anchor frames are extracted to form a map point set M={M i}, where M i Represents the set of map points of the i-th camera corresponding to the m anchor frames.

[0181] Step S103b: determine at least one map point from the map point set.

[0182] In the embodiment of the present application, the map point set includes multiple map points. In the embodiment of the present application, the at least one determined map point can be determined from the map point set according to a preset rule, or can be each map point in the map point set. This is not limited in the embodiment of the present application.

[0183] Step S103c: project at least one map point onto a corresponding camera plane, and extract a set of feature points satisfying a first preset condition from the current image based on the projection information.

[0184] For the embodiment of the present application, after obtaining at least one map point, the obtained at least one map point is projected onto the corresponding camera plane by formula (2), where formula (2) is:

[0185] Formula (2)

[0186] in, Represents the pixel coordinate point on the corresponding camera plane, represents the known camera model parameters, is a map point in the 3D space corresponding to the camera ∈ .

[0187] Further, after obtaining the pixel coordinate points projected onto each camera plane Then, search in The pixel coordinates are the feature points set Feat extracted from the current image within the circle with r1 as the radius and the center of the circle i , where the extracted feature point set Feat i is a set of feature points that meets the first preset condition.

[0188] Step S103d: based on the feature point set and the map point set that satisfy the first preset condition, determine the first matching pair set corresponding to each camera.

[0189] The first matching pair set corresponding to any camera includes: a plurality of first matching pairs, the first matching pairs are map points and feature points matched therewith, and the vector distance between the feature descriptor of the map point and the feature descriptor of the feature point matched therewith is the smallest.

[0190] Specifically, in getting Feat i Afterwards, calculate Saved feature descriptors and collections Feat i The distance between the feature points in the feature descriptor and the feature point p corresponding to the minimum descriptor distance i As map point P i The matching points of each camera are obtained, thereby obtaining the matching pair set {p i , } as the matching pair sets corresponding to each camera, that is, the first matching pair sets corresponding to each camera.

[0191] Step S103f, determining the initial pose of the initial anchor point based on the first matching pair set.

[0192] Specifically, after obtaining the matching pair sets {p i , }, based on the matching pair sets corresponding to each camera, the initial pose of the initial anchor point is determined by formula (3).

[0193] Among them, formula (3) is:

[0194] Formula (3)

[0195] in, is the initial pose of the initial anchor point.

[0196] Furthermore, after obtaining the initial pose of the initial anchor point, the initial pose of the current anchor point can be determined according to the initial pose of the initial anchor point. After obtaining the initial pose of the initial anchor point, the initial pose of the initial anchor point can be optimized to obtain the pose of the initial anchor point, so as to determine the pose of the current anchor point according to the pose of the initial anchor point. In the embodiment of the present application, the method of optimizing the initial pose of the initial anchor point to obtain the pose of the initial anchor point is detailed in the following steps S103g-S103j.

[0197] Step S103g: project at least one map point to a camera plane that meets a second preset condition.

[0198] The camera plane that meets the second preset condition is the camera plane that is closest to the camera corresponding to each map point in the clockwise direction in terms of Euclidean distance.

[0199] Specifically, after obtaining the initial pose of the initial anchor point After that, the camera cross-matching is performed, that is, the map points are matched by the above formula (2). Projected onto the camera plane of camera k that is closest to camera i in the Euclidean distance clockwise.

[0200] Among them, i≠k, 0≤k≤x, and x is the number of cameras.

[0201] Step S103h: extracting a set of feature points satisfying a third preset condition from the current image based on the projection information.

[0202] For the embodiment of the present application, the projection information is the pixel coordinate information of at least one map point in step S103 projected to the camera plane that meets the second preset condition, that is, the pixel coordinate point projected on the camera plane of camera k. After obtaining the pixel coordinate point projected on the camera plane of camera k, search for a set of feature points extracted on the current image within a circle with the pixel coordinate point as the center and r2 as the radius, and determine the extracted feature point set as a set of feature points that meets the third preset condition. In the embodiment of the present application, r2 and r1 may be the same or different, and are not limited in the embodiment of the present application.

[0203] Step S103i: Determine a second matching pair set based on the feature point set and the map point set that meet the third preset condition.

[0204] The second matching pair set is a matching pair set corresponding to cameras that meet the second preset condition.

[0205] For the embodiment of the present application, after obtaining a set of feature points that meet the third preset condition, the distance between the feature descriptor stored in the map point and the feature descriptor corresponding to the feature point in the set that meets the third preset condition is calculated to obtain the feature point corresponding to the minimum descriptor, and the feature point is determined as the matching point of the corresponding map point, thereby obtaining a second matching pair set. For example, the second matching pair set is {p k , }.

[0206] Step S103j: Based on the second matching pair set and the first matching pair sets corresponding to each camera, the initial pose of the initial anchor point is optimized to obtain the pose of the initial anchor point.

[0207] For the embodiment of the present application, after obtaining the second matching pair set {p k , } and the first matching pair set {p i , After that, based on this, the initial pose of the initial anchor point is optimized by the following formula (4) to obtain the pose of the initial anchor point.

[0208] Among them, formula (4) is

[0209] Formula (4);

[0210] in, is the pose of the initial anchor point; ∑ represents traversing the corresponding set of matching pairs.

[0211] Specifically, determining the posture of the current anchor point based on the posture of the initial anchor point in step S104 may include: step S1041 (not shown in the figure), step 1042 (not shown in the figure), step S1043 (not shown in the figure), step S1044 (not shown in the figure), step S1045 (not shown in the figure) and step S1046 (not shown in the figure), wherein:

[0212] Step S1041, obtaining the rigid body connection information between the current anchor point and the multiple cameras and the texture information of the multiple camera images corresponding to the current anchor point.

[0213] For the embodiment of the present application, the rigid body connection information between the current anchor point and the multiple cameras may include: the known multi-camera calibration algorithm can obtain the external parameter matrix of each camera relative to the anchor point coordinate system, marked as , where e represents extrinsic, Indicates the camera's tag number. , The number of cameras currently set.

[0214] Step S1042: Determine a plurality of third matching pair sets based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point.

[0215] Among them, any third matching pair set includes: a matching set between the first feature point corresponding to any camera connected to the current anchor point and the map point of the known map.

[0216] Specifically, based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point, multiple third matching pair sets are determined, which may specifically include: step S1042a (not shown in the figure), step S1042b (not shown in the figure), step S1042c (not shown in the figure), step S1042d (not shown in the figure), and step S1042e (not shown in the figure), wherein,

[0217] Step S1042a: Acquire a specific posture from a preset sensor.

[0218] For the embodiments of the present application, unmanned robots or unmanned vehicles may be equipped with external sensors, such as radar sensors, wheel speed odometers, and inertial sensors, in addition to multiple fisheye cameras; the specific posture may be a posture calculated based on the data measured by these external sensors, that is, the specific posture involved in the embodiments of the present application, for example, the specific posture may be referred to as T0.

[0219] Step S1042b: Based on the specific pose and the pose of the initial anchor point, determine the estimated pose corresponding to the anchor point frame at the current moment.

[0220] For the embodiment of the present application, after obtaining the specific pose T0 and the pose of the initial anchor point Afterwards, the estimated pose of the anchor frame at the current moment is calculated by formula (5), where formula (5) can be:

[0221] Formula (5)

[0222] in, is the estimated pose corresponding to the anchor frame at the current moment.

[0223] Step S1042c: Based on the estimated pose corresponding to the anchor frame at the current moment, determine the reference anchor frame from the known map.

[0224] The reference anchor point frame is the reference anchor point frame corresponding to the anchor point frame at the current moment.

[0225] For the embodiment of the present application, from the posture Get the current anchor point frame position in the known map ,in is the pose matrix The displacement vector part in . The distance between the anchor frame and the known map is calculated by formula (1), and the frame with the smallest Euclidean distance is selected as the reference frame of the current anchor frame.

[0226] Furthermore, after obtaining the reference anchor point frame, texture matching and geometric matching are performed based on the reference anchor point frame, wherein the texture matching method is detailed in step S1042d and step S1042e.

[0227] Step S1042d: Project the map points of each camera stored in the reference anchor frame to the camera image corresponding to the anchor frame at the current moment, and obtain a set of feature points corresponding to the anchor frame at the current moment.

[0228] Among them, the feature point set corresponding to the anchor point frame at the current moment is the feature point set corresponding to each camera respectively.

[0229] For the embodiment of the present application, the map point of the corresponding camera stored in the reference anchor point frame is projected to the camera image corresponding to the current anchor point frame by formula (2), and the feature points extracted from the current image are searched in the circle with a radius of r3 at the coordinates of the projection point to obtain the feature point set { In the embodiment of the present application, r3 may be the same as or different from at least one of r1 and r2, which is not limited in the embodiment of the present application.

[0230] Assume that there are four cameras, namely camera 1, camera 2, camera 3 and camera 4. Based on formula (2), the map points of camera 1 stored in the reference anchor frame are projected to the image of camera 1 corresponding to the current anchor frame. In the circle with radius r3 of the projection point coordinates, the feature points extracted from the current image are searched to obtain the feature point set { }.

[0231] Step S1042e: Determine each third matching pair set based on the feature descriptors stored in the known map and the feature descriptors corresponding to each feature point in the feature point set.

[0232] For the embodiment of the present application, each third matching pair set includes: a third matching pair set corresponding to each camera. In the embodiment of the present application, the feature descriptor stored in the known map point and the feature descriptor of the corresponding feature point in the feature point set are used to match and calculate the descriptor distance in sequence, and the feature point with the closest descriptor distance is the feature point corresponding to the map point, and the third matching pair set { },in, .

[0233] Step S1043: recursively extrapolate the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame.

[0234] Specifically, the matching information of the anchor frame at the last moment includes: each camera information corresponding to the anchor frame at the last moment, each camera information contains a matching set of corresponding feature points and map points. In the embodiment of the present application, the second feature point is each feature point other than the first feature point, and can also be a subset of each feature point other than the first feature point, which is not limited in the embodiment of the present application.

[0235] Specifically, in the embodiment of the present application, the matching information of the anchor frame at the previous moment is recursively transferred to the current anchor frame to obtain the predicted matching information of some feature points in the current anchor frame, which may specifically include: based on each matching set, predicting the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment. In the embodiment of the present application, the optical flow tracking algorithm is used to predict the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment, forming a set { }.

[0236] The positions of the feature points of the image at the previous moment in the camera image corresponding to the current moment are the positions of the respective feature points at the current moment in the camera image corresponding to the current moment.

[0237] Step S1044: Determine a plurality of fourth matching pair sets based on the predicted matching information of the second feature point in the current anchor point frame.

[0238] Wherein, any fourth matching pair set includes: a matching set between the second feature points corresponding to any camera and the map points of the known map.

[0239] Specifically, in step S1044, the fourth matching pair set is determined based on the predicted matching information of some feature points in the current anchor point frame, which may specifically include: determining the matching points corresponding to each feature point at the current moment based on the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment; constructing multiple fifth matching pair sets based on each feature point at the current moment and the matching points corresponding to each feature point at the current moment; and determining the corresponding fourth matching pair sets based on the multiple fifth matching pair sets.

[0240] For the embodiment of the present application, the set { }, the radius around it is Search for the feature points of the current image within the circle, calculate the Euclidean distance of its image coordinates, and the feature point with the smallest Euclidean distance is its matching point, forming the fifth matching pair set { , }, , and then according to its corresponding relationship , update the matching of the map points to the current frame, forming the fourth matching pair set { }.

[0241] Step S1045: merge and remove duplicates of each third matching pair set and each fourth matching pair set to obtain matching information of the current anchor point frame.

[0242] Specifically, the merged set { }as well as{ }, and remove duplicates to obtain the set of matching feature points of the current camera image and map points as { }, and the obtained set of matching of the current camera image feature points and the map points is determined as the matching information of the current anchor point. In the embodiment of the present application, the set { }as well as{ }The method of merging and deduplication is not limited in the embodiments of the present application.

[0243] Step S1046: Based on the matching information of the current anchor point frame, fitting processing is performed by the anchor point-based least squares method to obtain the position and posture of the current anchor point.

[0244] Specifically, in step S1046, the matching information of the current anchor frame is used to perform fitting processing based on the anchor point least square method to obtain the current anchor point posture. Specifically, it may include: based on the matching information of the current anchor frame, the estimated posture corresponding to the anchor frame at the current moment is optimized by the anchor point least square method to obtain the current anchor point posture. That is, after obtaining the matching set of feature points and map points of each camera of the anchor frame, the anchor frame posture is optimized by formula (6) to obtain the current anchor point posture. Among them,

[0245] Formula (6);

[0246] in, is the pose of the current anchor point.

[0247] Furthermore, after obtaining the pose of the current anchor point, in order to further improve the accuracy of the pose of the current anchor point, the pose of the current anchor point is determined based on the pose of the initial anchor point, and then the following steps may be included: step Sa (not shown in the figure) and step Sb (not shown in the figure), wherein:

[0248] Step Sa: determining matching information between the current anchor point frame and the local map based on a multi-combination matching method between the current anchor point frame and the local map.

[0249] Furthermore, in the embodiment of the present application, before step Sa, the following may be included: extracting a local map from a known map based on the position of the anchor frame at the current moment in the map. Specifically, in the embodiment of the present application, after obtaining the position and posture of the current anchor frame, Afterwards, from the pose of the current anchor frame Get the current anchor point frame position in the known map , calculate the distance between the anchor point frame and the known map through formula (1), and then sort the distance sequence from small to large, and take the first The anchor frame corresponding to the distance is used as the local map of the current anchor frame.

[0250] Further, after extracting the local map of the current anchor frame from the known map, the matching information of the current anchor frame and the local map is determined based on the multi-combination matching method of the current anchor frame and the local map, which may specifically include: step Sa1 (not shown in the figure), step Sa2 (not shown in the figure) and step Sa3 (not shown in the figure), wherein:

[0251] Step Sa1: projecting the map points corresponding to the cameras stored in the anchor frame of the local map to the cameras corresponding to the current anchor frame, and determining a plurality of sixth matching pair sets based on the projection information.

[0252] The multiple sixth matching pair sets include: sixth matching pair sets corresponding to each camera respectively.

[0253] Specifically, after projecting the map points corresponding to the cameras stored in the anchor frame of the local map to the cameras corresponding to the current anchor frame, multiple sixth matching pair sets are determined based on the projection information. In the embodiment of the present application, the map points of the corresponding cameras stored in the anchor frame of the local map are projected to the cameras corresponding to the current anchor frame, that is, the map points of the i-th camera in the anchor frame of the local map are projected to the i-th camera of the current anchor frame by formula (2). Similarly, in the circle with a radius of r4 at the coordinates of the projection point, the feature points extracted from the current image are searched to obtain the feature point set { }, c is the current anchor frame tag, and then the feature descriptor stored in the map point and the feature descriptor of the corresponding feature point in the feature point set are matched in sequence to calculate the descriptor distance, and the feature point with the closest descriptor distance is the feature point corresponding to the map point, and the sixth matching pair set { In the embodiment of the present application, the sixth matching pair set corresponding to each camera can be obtained in the above manner. In the embodiment of the present application, r4 can be the same as or different from at least one of r1, r2 and r3, which is not limited in the embodiment of the present application.

[0254] Step Sa2: Project the map points corresponding to each camera stored in the anchor frame of the local map to the first preset camera corresponding to the current anchor frame, and determine a plurality of seventh matching pair sets based on the projection information.

[0255] For the embodiment of the present application, the plurality of seventh matching pair sets include: matching pair sets corresponding to the respective cameras.

[0256] Specifically, in the embodiment of the present application, after projecting the map points corresponding to each camera stored in the anchor frame of the local map to the first preset camera corresponding to the current anchor frame, based on the projection information, the first preset camera corresponding to the current anchor frame is determined to be the camera with the closest Euclidean distance to each camera. In the embodiment of the present application, the map points corresponding to camera i stored in the anchor frame of the local map are projected to the camera k with the closest Euclidean distance to camera i in the current anchor frame, and the feature points extracted from the current image are searched in the circle with a radius of r5 at the same coordinates of the projection point to obtain the feature point set. , c is the current anchor frame tag, and then the feature descriptor stored in the map point and the feature descriptor of the corresponding feature point in the feature point set are matched in sequence to calculate the descriptor distance, and the feature point with the closest descriptor distance is the feature point corresponding to the map point, and the seventh matching pair set { }.

[0257] Step Sa3: Project the map points corresponding to each camera stored in the anchor frame of the local map to the second preset camera corresponding to the current anchor frame, and determine multiple eighth matching pair sets based on the projection information. In the embodiment of the present application, the second preset camera corresponding to the current anchor frame is the camera with the farthest Euclidean distance from each camera; the multiple eighth matching pair sets include: the eighth matching pair sets corresponding to each camera respectively.

[0258] For the embodiment of the present application, after projecting the map points corresponding to each camera stored in the anchor frame of the local map to the second preset camera corresponding to the current anchor frame, multiple eighth matching pair sets are determined based on the projection information. In the embodiment of the present application, the map points corresponding to camera i stored in the anchor frame of the local map are projected to the current anchor frame, and the map points of the i-th camera in the anchor frame of the local map are projected to the h-th camera in the current anchor frame through formula (2). Similarly, in the circle with a radius of r6 at the coordinates of the projection point, the feature points extracted from the current image are searched to obtain the feature point set { }, c is the current anchor frame tag, and then the feature descriptor stored in the map point and the feature descriptor of the corresponding feature point in the feature point set are used to match and calculate the descriptor distance in sequence, and the feature point with the closest descriptor distance is the feature point corresponding to the map point, and the eighth matching pair set {} is obtained. In the embodiment of the present application, r6 can be the same as or different from at least one of r1, r2, r3, r4 and r5, which is not limited in the embodiment of the present application.

[0259] It should be noted that the execution order of step Sa1, step Sa2 and step Sa3 is not limited. In the embodiment of the present application, step Sa1, step Sa2 and step Sa3 can be executed simultaneously, or any two of them can be executed simultaneously, or they can be executed at different times. When any two of step Sa1, step Sa2 and step Sa3 are executed simultaneously or at different times, the specific execution order is not limited in the embodiment of the present application.

[0260] Step Sb: Based on the matching information between the current anchor point frame and the local map, the position and posture of the initial anchor point are optimized to obtain the optimized position and posture of the current anchor point.

[0261] Specifically, on the basis of the above embodiments (step Sa1, step Sa2 and step Sa3), in step Sb, the pose of the initial anchor point is optimized based on the matching information between the current anchor point frame and the local map to obtain the optimized pose of the current anchor point, which may specifically include: based on multiple matching pair sets, multiple seventh matching pair sets and multiple eighth matching pair sets, the pose of the initial anchor point is optimized to obtain the optimized pose of the current anchor point.

[0262] Specifically, the current anchor point frame pose is optimized by formula (7) to obtain the optimized current anchor point pose, where formula (7) is:

[0263] .

[0264] in, is the optimized pose of the current anchor point.

[0265] The above embodiment introduces a relocation method from the perspective of method flow, and the following embodiment introduces a relocation device from the perspective of a virtual module. Please refer to the following embodiment for details.

[0266] The present application embodiment provides a relocation device, such as Figure 2 As shown, the relocation device 20 may include: a semantic matching module 21, a first determination module 22, a matching fitting module 23 and a second determination module 24, wherein:

[0267] The semantic matching module 21 is used to perform semantic matching on the multi-camera image acquired based on the initial anchor point and the multi-camera image anchored historically, so as to obtain the estimated position of the current anchor point in the known map.

[0268] The first determination module 22 is used to determine a local historical anchor point frame in a known map.

[0269] Among them, the local historical anchor point frame is the historical anchor point frame with the closest Euclidean distance to the estimated position.

[0270] The matching and fitting module 23 is used to perform multi-level matching and least square fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point;

[0271] The second determination module 24 is used to determine the position and posture of the current anchor point based on the position and posture of the initial anchor point.

[0272] In a possible implementation of the embodiment of the present application, the multi-camera image acquired based on the initial anchor point stores a feature descriptor corresponding to the initial anchor point, and the multi-camera image historically anchored stores a historical feature descriptor; when the semantic matching module 21 performs semantic matching on the multi-camera image acquired based on the initial anchor point and the multi-camera image historically anchored to obtain an estimated position of the current anchor point in a known map, it is specifically used to:

[0273] Determine the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point;

[0274] Determine the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor;

[0275] Calculate the vector distance between the semantic BOW of the current anchor point and the semantic BOW corresponding to each historical anchor point frame;

[0276] Determine the historical anchor point frame with the smallest vector distance;

[0277] Determine the position of the historical anchor point frame with the smallest vector in the known map as the estimated position of the current anchor point in the known map.

[0278] In another possible implementation of the embodiment of the present application, when the first determination module 22 determines the local historical anchor point frame in the map, it is specifically used to:

[0279] Based on the estimated position of the current anchor point in the known map and the positions of each historical anchor point frame in the known map, each Euclidean distance is determined, and each Euclidean distance is the Euclidean distance between the historical anchor point frame with the smallest vector distance and each historical anchor point frame;

[0280] The historical anchor frame with the smallest Euclidean distance is determined from each historical anchor frame as the local anchor frame.

[0281] In another possible implementation of the embodiment of the present application, the matching and fitting module 23 performs multi-level matching and least squares fitting on the feature points stored in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point, specifically for:

[0282] Extract the map points corresponding to each camera stored in the anchor frame with the smallest Euclidean distance and the historical anchor frame to obtain a map point set;

[0283] Determine at least one map point from the set of map points;

[0284] Projecting at least one map point onto a corresponding camera plane;

[0285] Extracting a set of feature points satisfying a first preset condition from the current image based on the projection information;

[0286] Based on a feature point set and a map point set that meet a first preset condition, determining a first matching pair set corresponding to each camera, wherein the first matching pair set corresponding to any camera includes: a plurality of first matching pairs, wherein the first matching pair is a map point and a feature point that matches the map point, and a vector distance between a feature descriptor of the map point and a feature descriptor of the feature point that matches the map point is the smallest;

[0287] Determine an initial pose of an initial anchor point based on the first set of matching pairs;

[0288] Projecting at least one map point to a camera plane that satisfies a second preset condition, where the camera plane that satisfies the second preset condition is a camera plane that has the shortest Euclidean distance in a clockwise direction to the camera corresponding to each map point;

[0289] Extracting a set of feature points satisfying a third preset condition from the current image based on the projection information;

[0290] Based on the feature point set and the map point set that meet the third preset condition, determine a second matching pair set, where the second matching pair set is a matching pair set corresponding to the cameras that meet the second preset condition;

[0291] Based on the first matching pair set and the second matching pair set corresponding to each camera, the initial pose of the initial anchor point is optimized to obtain the pose of the initial anchor point.

[0292] In another possible implementation of the embodiment of the present application, when the second determination module 24 determines the position and posture of the current anchor point based on the position and posture of the initial anchor point, it is specifically used to:

[0293] Get the rigid body connection information between the current anchor point and multiple cameras and the texture information of the multiple camera images corresponding to the current anchor point;

[0294] Based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point, each third matching pair set is determined, and any third matching pair set includes: a matching set of the first feature point corresponding to any camera connected to the current anchor point and the map point quality inspection of the map;

[0295] Recursively extrapolate the matching information of the previous anchor frame to the current anchor frame to obtain the predicted matching information of the second feature point in the current anchor frame;

[0296] Based on the predicted matching information of the second feature point in the current anchor point frame, a plurality of fourth matching pair sets are determined, wherein any fourth matching pair set includes: a matching set between the second feature point corresponding to any camera and a map point of a known map;

[0297] Merge and remove duplicates of each third matching pair set and the fourth matching pair set corresponding to each third matching pair set to obtain matching information of the current anchor point frame;

[0298] Based on the matching information of the current anchor point frame, the position and posture of the current anchor point are obtained by fitting through the anchor point-based least squares method.

[0299] In another possible implementation of the embodiment of the present application, the second determination module 24 is specifically used to determine multiple third matching pair matching sets based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point:

[0300] Get a specific pose from a preset sensor;

[0301] Based on the specific pose and the pose of the initial anchor point, determine the estimated pose corresponding to the anchor point frame at the current moment;

[0302] Based on the estimated pose corresponding to the anchor frame at the current moment, a reference anchor frame is determined from the known map, where the reference anchor frame is the reference anchor frame corresponding to the anchor frame at the current moment;

[0303] Project the map points corresponding to each corresponding camera stored in the reference anchor frame to the camera image corresponding to the anchor frame at the current moment, and obtain the feature point set corresponding to the anchor frame at the current moment, where the feature point set is the feature point set corresponding to each camera;

[0304] Based on the feature descriptors stored in the known map and the feature descriptors corresponding to the feature points in the feature point set, each third matching pair set is determined.

[0305] In another possible implementation of the embodiment of the present application, the matching information of the anchor frame at the previous moment includes: each camera information corresponding to the anchor frame at the previous moment, each camera information including a matching set of corresponding feature points and map points;

[0306] When the second determination module 24 recursively extrapolates the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame, it is specifically used to:

[0307] Based on each matching set, the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is predicted, and the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is the position of each feature point at the current moment in the camera image corresponding to the current moment.

[0308] In another possible implementation of the embodiment of the present application, when the second determination module 24 determines a plurality of fourth matching pair sets based on the predicted matching information of some feature points in the current anchor point frame, it is specifically configured to:

[0309] Based on the positions of the feature points of the image at the previous moment in the camera image corresponding to the current moment, the matching points corresponding to the feature points at the current moment are determined;

[0310] Constructing a plurality of fifth matching pair sets based on each feature point at the current moment and the matching points respectively corresponding to each feature point at the current moment;

[0311] The respective corresponding fourth matching pair sets are determined based on the plurality of fifth matching pair sets.

[0312] In another possible implementation of the embodiment of the present application, when the second determination module 24 performs fitting processing based on the matching information of the current anchor point frame and obtains the pose of the current anchor point by using the anchor point-based least squares method, it is specifically used to:

[0313] Based on the matching information of the current anchor point frame, the estimated pose corresponding to the anchor point frame at the current moment is optimized by the anchor point-based least squares method to obtain the pose of the current anchor point.

[0314] In another possible implementation of the embodiment of the present application, the device 20 further includes: a third determination module and an optimization module, wherein:

[0315] A third determination module is used to determine matching information between the current anchor point frame and the local map based on a multi-combination matching method between the current anchor point frame and the local map;

[0316] The optimization module is used to optimize the pose of the initial anchor point based on the matching information between the current anchor point frame and the local map to obtain the optimized pose of the current anchor point.

[0317] In another possible implementation manner of the embodiment of the present application, the position and posture of the current anchor point includes: the position of the anchor point frame at the current moment in the known map.

[0318] Furthermore, the device 20 further includes: an extraction module, wherein:

[0319] An extraction module, used to extract a local map from a known map based on the position of the anchor frame in the map at the current moment;

[0320] Wherein, when the third determination module determines the matching information between the current anchor point frame and the local map based on the multi-combination matching method between the current anchor point frame and the local map, it is specifically used to:

[0321] Project the map points corresponding to each camera stored in the anchor point frame of the local map to each camera corresponding to the current anchor point frame;

[0322] Based on the projection information, determining a plurality of sixth matching pair sets;

[0323] Projecting the map points corresponding to each camera stored in the anchor frame of the local map to the first preset camera corresponding to the current anchor frame, where the first preset camera corresponding to the current anchor frame is the camera with the closest Euclidean distance to each camera;

[0324] Based on the projection information, determining a plurality of seventh matching pair sets;

[0325] Projecting the map points corresponding to each camera stored in the anchor frame of the local map to the second preset camera corresponding to the current anchor frame, where the second preset camera corresponding to the current anchor frame is the camera with the farthest Euclidean distance from each camera;

[0326] Based on the projection information, a plurality of eighth matching pair sets are determined.

[0327] Another possible implementation method of the embodiment of the present application is that the optimization module optimizes the posture of the initial anchor point based on the matching information of the current anchor point frame and the local map to obtain the optimized posture of the current anchor point, and is specifically used to: optimize the posture of the initial anchor point based on multiple sixth matching pair sets, multiple seventh matching pair sets, and multiple eighth matching pair sets to obtain the optimized posture of the current anchor point.

[0328] It should be noted that the first determination module 22 and the second determination module 24 may be the same determination module or different determination modules, which is not limited in the embodiment of the present application.

[0329] The embodiment of the present application provides a repositioning device. Compared with the related art in which the initial pose during repositioning is calculated by combining the GP3P algorithm with the UPNP algorithm, in the embodiment of the present application, semantic matching is performed between the multi-camera image obtained by the initial anchor point and the multi-camera image of the historical anchor to obtain the estimated position of the current anchor point in the known map, and the local historical anchor point frame can be determined in the known map according to the estimated position, and then the feature points stored in the local historical anchor point frame are respectively matched with the feature points extracted from each camera image by multi-level matching and least squares fitting to obtain the pose of the initial anchor point, so as to further determine the pose of the current anchor point according to the pose of the initial anchor point. That is, in the embodiment of the present application, the pose of the initial anchor point is obtained by the above calculation as prior information, and then the pose of the current anchor point (initial pose during repositioning) is calculated based on the prior information, so that even in the case of relatively large noise, the accuracy of the obtained initial pose during repositioning can be improved.

[0330] An electronic device is provided in an embodiment of the present application, such as Figure 3 As shown, Figure 3 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0331] The processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0332] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0333] The memory 303 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0334] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0335] The electronic devices include, but are not limited to, mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. In the embodiment of the present application, the server can be a cloud server. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0336] The embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer is run on a computer, the computer can execute the corresponding content of the aforementioned method embodiment. Compared with the related art, in the embodiment of the present application, the multi-camera image obtained by the initial anchor point is semantically matched with the multi-camera image of the historical anchor to obtain the estimated position of the current anchor point in the known map, and the local historical anchor point frame can be determined in the known map according to the estimated position, and then the feature points stored in the local historical anchor point frame are respectively matched with the feature points extracted from each camera image for multi-level matching and least squares fitting to obtain the pose of the initial anchor point, so as to further determine the pose of the current anchor point according to the pose of the initial anchor point, that is, in the embodiment of the present application, the pose of the initial anchor point is obtained by the above calculation as prior information, and then the pose of the current anchor point (initial pose during relocation) is calculated based on the prior information, so that even in the case of relatively large noise, the accuracy of the initial pose obtained during relocation can be improved.

[0337] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0338] In the embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0339] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0340] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0341] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks or optical disks.

[0342] As described above, the above embodiments are only used to introduce the technical solution of the present application in detail, but the description of the above embodiments is only used to help understand the method and core idea of ​​the present application and should not be understood as limiting the present application. Any changes or substitutions that can be easily thought of by technicians in this technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. A relocation method, characterized in that: include: Semantically match the multi-camera images acquired based on the initial anchor point with the multi-camera images of the historical anchor point to obtain the estimated position of the current anchor point in the known map; Determining a local historical anchor point frame in the known map, the local historical anchor point frame being a historical anchor point frame having a closest Euclidean distance to the estimated position; Perform multi-level matching and least squares fitting on the feature points saved in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and pose of the initial anchor point; Determine the pose of the current anchor point based on the pose of the initial anchor point; The method of determining the position and posture of the current anchor point based on the position and posture of the initial anchor point includes: obtaining rigid connection information between the current anchor point and multiple cameras and texture information of the multiple camera images corresponding to the current anchor point; determining each third matching pair set based on the rigid connection information between the current anchor point and multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point and the position and posture of the initial anchor point, wherein any third matching pair set includes: a matching set between the first feature point corresponding to any camera connected to the current anchor point and the map point of the known map; recursively extrapolating the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame; determining multiple fourth matching pair sets based on the predicted matching information of the second feature point in the current anchor point frame, wherein any fourth matching pair set includes: a matching set between the second feature point corresponding to any camera and the map point of the known map; merging and deduplicating each third matching pair set and the corresponding fourth matching pair set to obtain the matching information of the current anchor point frame; and obtaining the position and posture of the current anchor point based on the matching information of the current anchor point frame and fitting by the anchor-based least squares method.

2. The method according to claim 1, characterized in that The multi-camera image acquired based on the initial anchor point stores a feature descriptor corresponding to the initial anchor point, and the multi-camera image historically anchored stores a historical feature descriptor; the multi-camera image acquired based on the initial anchor point is semantically matched with the multi-camera image historically anchored to obtain an estimated position of the current anchor point in the known map, including: Determine the semantic BOW of the initial anchor point based on the feature descriptor corresponding to the initial anchor point; Determine the semantic BOW corresponding to each historical anchor point frame based on the historical feature descriptor; Calculate the vector distances between the semantic BOW of the current anchor point and the semantic BOWs corresponding to each of the historical anchor point frames; Determine the historical anchor point frame with the smallest vector distance; The position of the historical anchor point frame with the smallest vector in the known map is determined as the estimated position of the current anchor point in the known map.

3. The method according to claim 2, characterized in that The determining of a local historical anchor point frame in the known map comprises: Based on the estimated position of the current anchor point in the known map and the positions of each historical anchor point frame in the known map, determine each Euclidean distance, wherein each Euclidean distance is the Euclidean distance between the historical anchor point frame with the smallest vector distance and each historical anchor point frame; A historical anchor point frame with the smallest Euclidean distance is determined from the historical anchor point frames as a local anchor point frame.

4. The method according to claim 3, characterized in that The feature points stored in the local historical anchor point frame are matched with the feature points extracted from each camera image by multi-level matching and least squares fitting to obtain the position and posture of the initial anchor point, including: Extracting the map points corresponding to each camera stored in the anchor point frame with the smallest Euclidean distance and the historical anchor point frame to obtain a map point set; determining at least one map point from the set of map points; Projecting the at least one map point onto a corresponding camera plane respectively; Extracting a set of feature points satisfying a first preset condition from the current image based on the projection information; Based on the feature point set that meets the first preset condition and the map point set, a first matching pair set corresponding to each camera is determined, and the first matching pair set corresponding to any camera includes: A plurality of first matching pairs, wherein the first matching pairs are a map point and a feature point matched therewith, and a vector distance between a feature descriptor of the map point and a feature descriptor of the feature point matched therewith is the smallest; Determining an initial pose of the initial anchor point based on the first set of matching pairs; Projecting the at least one map point to a camera plane that satisfies a second preset condition, wherein the camera plane that satisfies the second preset condition is a camera plane that is closest to the camera corresponding to each map point in a clockwise Euclidean distance; Extracting a set of feature points satisfying a third preset condition from the current image based on the projection information; Based on the feature point set that meets the third preset condition and the map point set, determining a second matching pair set, where the second matching pair set is a matching pair set corresponding to the camera that meets the second preset condition; Based on the first matching pair set and the second matching pair set respectively corresponding to each camera, the initial pose of the initial anchor point is optimized to obtain the pose of the initial anchor point.

5. The method according to claim 1, characterized in that The determining of a plurality of third matching pair sets based on the rigid body connection information between the current anchor point and the multiple cameras, the texture information of the multiple camera images corresponding to the current anchor point, and the position and posture of the initial anchor point comprises: Get a specific pose from a preset sensor; Based on the specific pose and the pose of the initial anchor point, determining an estimated pose corresponding to the anchor point frame at the current moment; Based on the estimated position and posture corresponding to the anchor frame at the current moment, determining a reference anchor frame from a known map, wherein the reference anchor frame is a reference anchor frame corresponding to the anchor frame at the current moment; Projecting the map points corresponding to each camera stored in the reference anchor frame to the camera image corresponding to each anchor frame at the current moment, to obtain a feature point set corresponding to the anchor frame at the current moment, wherein the feature point set is a feature point set corresponding to each camera; The third matching pair sets are determined based on the feature descriptors stored in the known map and the feature descriptors corresponding to the feature points in the feature point set.

6. The method according to claim 5, characterized in that The matching information of the anchor frame at the previous moment includes: The camera information corresponding to the anchor point frame at the previous moment, each camera information including a matching set of corresponding feature points and map points; The step of recursively extending the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame includes: Based on each matching set, the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is predicted, and the position of the feature points of the image at the previous moment in the camera image corresponding to the current moment is the position of each feature point at the current moment in the camera image corresponding to the current moment.

7. The method according to claim 6, characterized in that The determining of a plurality of fourth matching pair sets based on the predicted matching information of some feature points in the current anchor point frame comprises: Based on the positions of the feature points of the image at the previous moment in the camera image corresponding to the current moment, the matching points corresponding to the feature points at the current moment are determined; constructing a plurality of fifth matching pair sets based on each feature point at the current moment and the matching points respectively corresponding to each feature point at the current moment; The respective corresponding fourth matching pair sets are determined based on the plurality of fifth matching pair sets.

8. The method according to any one of claims 5 to 7, characterized in that: The matching information of the current anchor point frame is matched based on the anchor point frame, and the pose of the current anchor point is obtained by fitting the matching information based on the anchor point frame, including: Based on the matching information of the current anchor point frame, the estimated position and posture corresponding to the anchor point frame at the current moment is optimized by using the anchor point-based least squares method to obtain the position and posture of the current anchor point.

9. The method according to claim 1, characterized in that: The step of determining the position and posture of the current anchor point based on the position and posture of the initial anchor point further comprises: Determine matching information between the current anchor point frame and the local map based on a multi-combination matching method between the current anchor point frame and the local map; Based on the matching information between the current anchor point frame and the local map, the position and posture of the initial anchor point are optimized to obtain the optimized position and posture of the current anchor point.

10. The method according to claim 9, characterized in that The position and posture of the current anchor point include: The position of the anchor frame at the current moment in the known map; The method of determining the matching information between the current anchor point frame and the local map based on the multi-combination matching between the current anchor point frame and the local map also includes: Extracting a local map from the known map based on the position of the anchor frame at the current moment in the map; The method of determining the matching information between the current anchor point frame and the local map based on the multi-combination matching between the current anchor point frame and the local map includes: Projecting the map points corresponding to the cameras stored in the anchor point frame of the local map to the cameras corresponding to the current anchor point frame; Based on the projection information, determining a plurality of sixth matching pair sets; Projecting the map points corresponding to each camera stored in the anchor point frame of the local map to the first preset camera corresponding to the current anchor point frame, wherein the first preset camera corresponding to the current anchor point frame is the camera with the closest Euclidean distance to each camera; Based on the projection information, determining a plurality of seventh matching pair sets; Projecting the map point corresponding to each camera stored in the anchor point frame of the local map to the second preset camera corresponding to the current anchor point frame, wherein the second preset camera corresponding to the current anchor point frame is the camera with the farthest Euclidean distance from each camera; Based on the projection information, a plurality of eighth matching pair sets are determined.

11. The method according to claim 10, characterized in that The step of optimizing the position and posture of the initial anchor point based on the matching information between the current anchor point frame and the local map to obtain the optimized position and posture of the current anchor point includes: Based on the multiple sixth matching pair sets, the multiple seventh matching pair sets, and the multiple eighth matching pair sets, the pose of the initial anchor point is optimized to obtain an optimized pose of the current anchor point.

12. A relocation device, characterized in that: include: The semantic matching module is used to semantically match the multi-camera images acquired based on the initial anchor point with the multi-camera images of the historical anchor point to obtain the estimated position of the current anchor point in the known map; A first determination module is used to determine a local historical anchor point frame in the known map, wherein the local historical anchor point frame is a historical anchor point frame with the shortest Euclidean distance to the estimated position; The matching and fitting module is used to perform multi-level matching and least squares fitting on the feature points saved in the local historical anchor point frame and the feature points extracted from each camera image to obtain the position and posture of the initial anchor point; A second determination module, configured to determine the position and posture of the current anchor point based on the position and posture of the initial anchor point; The second determination module is also used to obtain the rigid connection information between the current anchor point and multiple cameras and the texture information of the multi-camera images corresponding to the current anchor point; based on the rigid connection information between the current anchor point and multiple cameras, the texture information of the multi-camera images corresponding to the current anchor point and the position and posture of the initial anchor point, determine each third matching pair set, any third matching pair set includes: a matching set between the first feature point corresponding to any camera connected to the current anchor point and the map point of the known map; recursively push the matching information of the anchor point frame at the previous moment to the current anchor point frame to obtain the predicted matching information of the second feature point in the current anchor point frame; based on the predicted matching information of the second feature point in the current anchor point frame, determine multiple fourth matching pair sets, any fourth matching pair set includes: a matching set between the second feature point corresponding to any camera and the map point of the known map; merge and remove duplicates of each third matching pair set and the corresponding fourth matching pair set to obtain the matching information of the current anchor point frame; based on the matching information of the current anchor point frame, and fitting processing is performed by the anchor-based least squares method to obtain the position and posture of the current anchor point.

13. An electronic device, characterized in that: It includes: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to: execute the relocation method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the relocation method as described in any one of claims 1 to 11.

Images