Multi-channel camera segmented positioning method, device, electronic equipment and storage medium

Abstract

The present disclosure also provides a multi-channel camera segmented positioning method, device, electronic device, and readable storage medium. The multi-channel camera segmented positioning method provided by the embodiments of the present disclosure includes: determining an optimal subgraph based on external sensor data and prior visual map data; performing positioning through sub-channel camera image matching and multi-channel camera image cross-matching, utilizing a map frame database corresponding to the optimal subgraph and a first pose obtained based on external sensor data to obtain a second pose; and enabling or disabling the positioning function based on the second pose by evaluating trajectory distance information and / or image texture information. The embodiments of the present disclosure can achieve robust, continuous, and high-precision positioning with low computing power consumption.

Description

Technical Field

[0001] The present disclosure relates to a multi-channel camera segmented positioning method, device, electronic equipment and storage medium. Background Art

[0002] When a robot moves in an unknown environment, it can use Simultaneous Localization and Mapping (SLAM) technology to locate itself based on its own measurement data and external observation data, and perform incremental mapping at the same time. The constructed SLAM map can be used as a priori map to improve the positioning effect of the robot when it moves again in the environment.

[0003] Visual maps are the most commonly used form of prior maps in positioning technology, offering advantages such as low cost, high accuracy, strong reusability, and strong updatability. However, visual maps are rich in information, and their reuse places high demands on both computing power and memory. Furthermore, complex trajectories and varying degrees of scene information richness lead to varying degrees of positioning difficulty, necessitating appropriate and appropriate allocation of computing resources in different situations. Furthermore, application scenarios with high positioning difficulty and a high probability of mispositioning, such as large-angle turns and areas with repetitive textures, place higher demands on the robustness of the positioning algorithm. Therefore, a robust, continuous, and high-precision positioning solution with low computing power consumption is required. Summary of the Invention

[0004] In order to solve at least one of the above technical problems, the present disclosure provides a multi-camera segmented positioning method, device, electronic device and storage medium.

[0005] According to a first aspect of the present disclosure, a multi-camera segmented positioning method is provided, comprising:

[0006] Determine the optimal submap based on data from external sensors and prior visual map data;

[0007] By performing image matching of separate cameras and cross-matching of images of multiple cameras, positioning is performed using a map frame database corresponding to the optimal sub-image and a first pose obtained based on data from an external sensor to obtain a second pose;

[0008] Based on the second pose, the localization function is turned on or off by evaluating the trajectory distance information and / or the image texture information.

[0009] In some possible implementations of the first aspect of the present disclosure, determining the optimal submap based on data from external sensors and prior visual map data includes:

[0010] According to the running trajectory and positioning situation, one of the continuous positioning mode and the jump positioning mode is used to determine the optimal subgraph required for system positioning;

[0011] The continuous positioning mode includes: determining the next sub-graph of the current sub-graph as the optimal sub-graph;

[0012] The jump positioning mode includes: determining the similar frame set in each subgraph using a DBoW-based bag-of-words model based on a global keyframe database in the prior visual map data, calculating the comprehensive similarity, and determining the subgraph with the highest comprehensive similarity as the optimal subgraph.

[0013] In some possible implementations of the first aspect of the present disclosure, based on the running trajectory and positioning conditions, one of the continuous positioning mode and the jump positioning mode is adopted to determine the optimal sub-graph required for system positioning, including: updating the number of sub-graphs that have failed to be positioned continuously based on the positioning conditions of the previous sub-graph; if the number of sub-graphs that have failed to be positioned continuously is less than or equal to a first number threshold, selecting the continuous positioning mode to determine the optimal sub-graph; if the number of sub-graphs that have failed to be positioned continuously is greater than the first number threshold, selecting the jump positioning mode to determine the optimal sub-graph.

[0014] In some possible implementations of the first aspect of the present disclosure, the determining of the optimal subgraph based on the data of the external sensor and the prior visual map data further includes: when the system is initialized, determining the first position of the starting point based on the data of the external sensor, searching for the corresponding subgraph within a predetermined range of the first position of the starting point based on the global subgraph segment endpoint library in the prior visual map data, importing the key frame database of the corresponding subgraph, using the DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculating the comprehensive similarity, and taking the subgraph with the highest comprehensive similarity as the optimal subgraph of the starting point.

[0015] In some possible implementations of the first aspect of the present disclosure, the determining of the optimal submap based on data from external sensors and prior visual map data also includes: when the positioning function is turned on, using one of the continuous positioning mode and the jump positioning mode to re-determine the optimal submap required for system positioning.

[0016] In some possible implementations of the first aspect of the present disclosure, the method of performing positioning by matching branch camera images and cross-matching multi-channel camera images, and using the map frame database corresponding to the optimal sub-image and the first pose obtained based on data from external sensors to obtain the second pose, includes: using a positioning algorithm based on a multi-camera adaptive matching method of rigid body combination to perform branch camera image matching and cross-matching multi-channel camera images, and using the map frame database corresponding to the optimal sub-image and the first pose obtained based on data from external sensors to obtain the second pose.

[0017] In some possible implementations of the first aspect of the present disclosure, the positioning is performed by split-channel camera image matching and multi-channel camera image cross-matching, using the map frame database corresponding to the optimal sub-graph and the first pose obtained based on data from external sensors to obtain a second pose, including: determining the initial pose of the current anchor point based on the positioning status of the previous anchor point and the relationship between the current anchor point and the sub-graph initial frame of the optimal sub-graph; optimizing the initial pose of the current anchor point by split-channel camera image matching to obtain the first optimized pose of the current anchor point; optimizing the first optimized pose of the current anchor point by multi-channel camera image cross-matching to obtain the positioning pose of the current anchor point; updating the visually corrected odometer according to the positioning pose of the current anchor point and the first pose obtained based on data from external sensors, and the updated visually corrected odometer pose is the second pose.

[0018] In some possible implementations of the first aspect of the present disclosure, the method of turning on or off the positioning function by evaluating trajectory distance information and / or image texture information according to the second posture includes: when the system is currently in a positioning state, determining whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the end frame of the sub-image according to the second posture and a pre-set first distance threshold.

[0019] In some possible implementations of the first aspect of the present disclosure, the positioning function is turned on or off by evaluating the trajectory distance information and / or image texture information according to the second posture, including: turning off the positioning function when the trajectory distance from the current anchor point to the sub-image end frame is less than the first distance threshold.

[0020] In some possible implementations of the first aspect of the present disclosure, the positioning function is turned on or off by evaluating the trajectory distance information and / or image texture information according to the second posture, including: when the system is currently in a non-positioning state, according to the second posture, the first posture obtained based on data from an external sensor, and a pre-set second distance threshold and fourth quantity threshold, by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image, to determine whether to turn on the positioning function.

[0021] In some possible implementations of the first aspect of the present disclosure, enabling or disabling the positioning function by evaluating trajectory distance information and / or image texture information according to the second posture includes:

[0022] Enable the positioning function in one of the following situations:

[0023] The trajectory distance from the current anchor point to the starting point frame of the next sub-image is less than the second distance threshold;

[0024] The number of cameras in the valid camera set whose gray level co-occurrence matrix information entropy of the camera image corresponding to the current anchor point is greater than the fourth quantity threshold is greater than the fourth quantity threshold.

[0025] According to a second aspect of the present disclosure, a multi-channel camera segmented positioning device is provided, comprising:

[0026] A submap switching module is used to determine the optimal submap based on the data from external sensors and the prior visual map data;

[0027] A multi-camera relocalization and tracking module is used to obtain a second pose by performing image matching of separate cameras and cross-matching of images of multiple cameras, using a map frame database corresponding to the optimal sub-image and a first pose obtained based on data from external sensors;

[0028] The positioning segmentation module is used to enable or disable the positioning function by evaluating the trajectory distance information and / or the image texture information according to the second posture.

[0029] In some possible implementations of the second aspect of the present disclosure, the subgraph switching module is specifically used to determine the optimal subgraph required for system positioning by adopting one of the continuous positioning mode and the jump positioning mode according to the running trajectory and positioning conditions; the continuous positioning mode includes: determining the next subgraph of the current subgraph as the optimal subgraph; the jump positioning mode includes: determining the set of similar frames in each subgraph according to the global key frame database in the prior visual map data, using the DBoW-based bag-of-words model, calculating the comprehensive similarity, and determining the subgraph with the highest comprehensive similarity as the optimal subgraph.

[0030] In some possible implementations of the second aspect of the present disclosure, the subgraph switching module is specifically used to determine the first position of the starting point according to the data of the external sensor when the system is initialized, search for the corresponding subgraph within a predetermined range of the first position of the starting point according to the global subgraph segment endpoint library in the prior visual map data, import the key frame database of the corresponding subgraph, use the DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculate the comprehensive similarity, and use the subgraph with the highest comprehensive similarity as the optimal subgraph of the starting point.

[0031] In some possible implementations of the second aspect of the present disclosure, the sub-image switching module is specifically configured to, when the positioning function is enabled, adopt one of a continuous positioning mode and a jump positioning mode to re-determine an optimal sub-image required for system positioning.

[0032] In some possible implementations of the second aspect of the present disclosure, the multi-camera relocalization and tracking module is specifically used to perform branch camera image matching and multi-camera image cross-matching using a positioning algorithm based on a multi-camera adaptive matching method of a rigid body combination method, and to perform positioning using the map frame database corresponding to the optimal sub-graph and the first pose obtained based on data from an external sensor, thereby obtaining the second pose.

[0033] In some possible implementations of the second aspect of the present disclosure, the multi-camera relocalization and tracking module is specifically used to: determine the initial pose of the current anchor point based on the positioning of the previous anchor point and the relationship between the current anchor point and the initial frame of the sub-image of the optimal sub-image; optimize the initial pose of the current anchor point through branch camera image matching to obtain the first optimized pose of the current anchor point; optimize the first optimized pose of the current anchor point through multi-camera image cross-matching to obtain the positioning pose of the current anchor point; update the visually corrected odometer according to the positioning pose of the current anchor point and the first pose obtained based on data from external sensors, and the pose of the updated visually corrected odometer is the second pose.

[0034] In some possible implementations of the second aspect of the present disclosure, the positioning segmentation module is specifically used to: when the system is currently in a positioning state, determine whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the end frame of the sub-image based on the second posture and a pre-set first distance threshold.

[0035] In some possible implementations of the second aspect of the present disclosure, the positioning segmentation module is specifically configured to disable the positioning function when the trajectory distance from the current anchor point to the end frame of the sub-image is less than the first distance threshold.

[0036] In some possible implementations of the second aspect of the present disclosure, the positioning segmentation module is specifically used to: when the system is currently in a non-positioning state, determine whether to turn on the positioning function by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image based on the second posture, the first posture obtained based on data from the external sensor, and the pre-set second distance threshold and fourth quantity threshold.

[0037] In some possible implementations of the second aspect of the present disclosure, the positioning segmentation module is specifically used to: turn on the positioning function in one of the following cases: the trajectory distance from the current anchor point to the starting frame of the next sub-image is less than the second distance threshold; the grayscale co-occurrence matrix information entropy of the camera image corresponding to the current anchor point is greater than the fourth quantity threshold, and the number of cameras in the valid camera set is greater than the fourth quantity threshold.

[0038] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0039] a memory storing execution instructions; and

[0040] A processor executes the execution instructions stored in the memory, so that the processor performs the above-mentioned multi-camera segmented positioning method.

[0041] According to a fourth aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned multi-camera segmented positioning method.

[0042] In the disclosed embodiment, positioning timing is controlled based on distance and texture. Only a single sub-image is loaded during positioning. When determining to load a sub-image, the required sub-image is determined in real time based on the trajectory and positioning conditions. During the positioning process, an adaptive matching method of multi-channel camera branch camera image matching and multi-channel camera image cross-matching is adopted for positioning, thereby achieving robust, continuous, and high-precision positioning with low computing power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0044] Figure 1 4 is a flow chart of a multi-camera segmentation method according to an embodiment of the present disclosure.

[0045] Figure 2 The present invention is a flowchart of determining an optimal subgraph in a multi-camera segmentation method according to an embodiment of the present invention.

[0046] Figure 3 FIG. 4 is a schematic diagram of an exemplary flow of positioning in a multi-camera segmentation method according to an embodiment of the present disclosure.

[0047] Figure 4 It is a schematic block diagram of the structure of a multi-camera segmented positioning device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0048] Description of Reference Numerals

[0049] 400 multi-channel camera segment positioning device

[0050] 500 bus

[0051] 600 processor

[0052] 700 Memory

[0053] 800 various other circuits DETAILED DESCRIPTION

[0054] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0055] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] Unless otherwise stated, the exemplary embodiments / examples shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of the various embodiments / examples may be further combined, separated, interchanged, and / or rearranged without departing from the technical concepts of the present disclosure.

[0057] The use of cross hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise indicated, the presence or absence of cross hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the order described. In addition, the same figure numbers represent the same components.

[0058] When a component is referred to as being “on,” “over,” “connected to,” or “coupled to” another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being “directly on,” “directly connected to,” or “directly coupled to” another component, there are no intervening components present. For this purpose, the term “connected” may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0059] The terms used herein are for the purpose of describing specific embodiments and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, the features, integral bodies, steps, operations, parts, assemblies and / or their groups stated are indicated, but the presence or addition of one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups is not excluded. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, and as such, they are used to explain the inherent deviations of the measured values, calculated values ​​and / or values ​​provided that will be recognized by those of ordinary skill in the art.

[0060] The following is a brief description of existing positioning technologies.

[0061] Related Technology 1: Patent No. CN 113537171 A, "A Method for Segmenting SLAM Maps," involves a method for segmenting SLAM maps. This method selects segmented locations based on trajectory information, map size, and positioning locations, dividing the map into multiple submaps. When using these submaps as prior positioning submaps, only the submaps need to be loaded, eliminating the need to load the entire map. This method reduces the memory usage and computing power consumption of the positioning map, while also reducing the risk of lost positioning when switching maps. However, this method requires overlap between submaps and only involves the map segmentation method, not the positioning algorithm.

[0062] Related Technology 2: Patent No. CN 109540148 A, "SLAM Map-Based Positioning Method and System," relates to a SLAM map-based positioning method that establishes a pose correspondence between a vehicle positioning module and the SLAM map, thereby determining the vehicle's position within the SLAM map. This method improves the accuracy of vehicle positioning but does not involve visual positioning algorithms.

[0063] Related Technology 3: Patent No. CN 113051424 A, "A Positioning Method and Device Based on SLAM Maps," relates to a positioning method and device based on SLAM maps. This method uses point cloud data from 2D images and posture information to construct a SLAM map. The device's posture is then determined by feature matching of newly acquired 2D images of the surrounding environment based on the SLAM map. This method uses visual matching to locate 2D images within the SLAM map, but does not involve multi-camera visual positioning methods.

[0064] Related Technology 4: Patent No. CN 105869136 A, "A Collaborative Visual SLAM Method Based on Multiple Cameras," relates to a collaborative visual SLAM method using multiple cameras in dynamic environments. This method performs pose estimation, map point classification, and camera group management on multiple independently movable cameras to reconstruct the three-dimensional trajectory of moving objects in dynamic scenes. This method is more robust than SLAM methods based on a single camera, but its multiple movable cameras make it more suitable for micro-robots and wearable augmented reality devices, rather than rigidly coupled multi-camera systems such as autonomous driving.

[0065] Related Technology 5: Patent No. CN 114742886 A, "A Repositioning Method, Apparatus, Electronic Device, and Storage Medium," relates to a multi-camera repositioning method that uses least squares fitting to obtain an initial pose. This method then adjusts the multi-camera matching method to achieve multi-level matching, completing the repositioning of the multi-camera system. This method fixes the extrinsic parameters of the multiple cameras relative to the anchor point and introduces a priori poses for repositioning, reducing the impact of noise. The multi-level matching method improves the success rate and accuracy of repositioning. However, it fails to account for mismatches caused by repeated texture areas and information imbalances between multiple cameras.

[0066] Related technologies 3-5 use visual maps as prior maps to localize newly acquired images, improving the success rate of localization. However, visual maps are rich in information, and localization within a global map requires more memory and computing power. Related technology 1 divides the visual map by evaluating keyframe positions, information content, and map size, and supports single sub-maps for localization, which reduces the resource consumption of the prior visual map to a certain extent. However, it requires overlapping sub-maps and suffers from low efficiency and success rate in texture-deficient areas where localization is difficult. Related technology 3 uses a monocular camera's 2D image library and pose information to construct a point cloud map, but suffers from the limitations of a small camera field of view and insufficient scene information. Related technologies 4-5 use multi-camera localization, which offers improved robustness compared to a monocular camera. However, the multi-camera approach in Related technology 4 can move relatively independently, making it more suitable for microrobots and wearable devices. Related technology 5 uses a multi-camera multi-level matching approach, which improves both accuracy and robustness, but its accuracy and success rate are still limited in complex scenes and trajectories, such as areas with repeated textures and trajectories with large yaw angles.

[0067] In view of this, the embodiments of the present disclosure provide the following segmented positioning method, device, electronic device and storage medium based on multiple cameras. Figures 1 to 4 The specific implementation of the embodiment of the present disclosure is described in detail.

[0068] It should be noted that the "position and posture" in the embodiments of the present disclosure includes position information and posture information. The position information may be three-dimensional coordinates in an absolute coordinate system, and the posture information may include multi-dimensional posture angles in the absolute coordinate system. The embodiments of the present disclosure do not limit the specific representation of the position and posture information, nor their dimensions.

[0069] The embodiments of the present disclosure are applicable to various scenarios or objects requiring real-time positioning. For example, the embodiments of the present disclosure are applicable to autonomous driving scenarios, logistics positioning scenarios, intelligent robot positioning scenarios, and drone positioning scenarios. The objects that can be positioned using the embodiments of the present disclosure include, but are not limited to, various vehicles, logistics vehicles, intelligent robots, drones, and the like.

[0070] Figure 1 FIG. 1 is a flow chart showing a multi-camera segmented positioning method according to some embodiments of the present disclosure. Figure 1 As shown, the exemplary process S10 of the multi-camera segmented positioning method may include the following steps:

[0071] Step S12, determining the optimal subgraph based on the data from the external sensor and the prior visual map data;

[0072] In some embodiments, the data of the external sensor may include a first position or a first posture determined based on data collected by the external sensor and / or based on data provided by the external sensor. The first posture may include a first position and a first posture. The first position may be an absolute coordinate in an absolute coordinate system, and the first posture may be posture information in an absolute coordinate system.

[0073] In some embodiments, the external sensor may be, but is not limited to, a wheel speed meter, an inertial measurement unit (IMU), a global positioning system (GPS), or other devices, used alone or in combination. In specific applications, the external sensor may be mounted on a positioning object such as a vehicle or robot, so that the first position or first pose of the positioning object can be obtained in real time through the external sensor.

[0074] In some implementations, the prior visual map data may include a global keyframe database and a global subgraph segment endpoint library that record global information, and a segment subgraph set that records local information.

[0075] The global keyframe database records the correspondence between word bag information and keyframes in the form of an inverted index. Each index records the serial number of a word in the word bag and the serial numbers of all keyframes containing the word. At the same time, each keyframe records the serial number of the subgraph to which it belongs. The global subgraph segment endpoint library records the starting keyframe serial number, position (for example, absolute coordinates in an absolute coordinate system) and serial number of the subgraph to which it belongs of each segmented subgraph. The segmented subgraph set is a set formed by multiple subgraphs stored in the order of motion trajectories when the map is constructed. Each subgraph in the segmented subgraph set stores the starting keyframe serial number and position information of its next subgraph (for example, absolute coordinates in an absolute coordinate system). The "subgraph starting frame" below refers to the "starting keyframe of a single-segment subgraph" and the "subgraph ending frame" refers to the "ending keyframe of a single-segment subgraph".

[0076] In some embodiments, step S12 may include determining the optimal submap required for system positioning based on the running trajectory and positioning conditions using one of a continuous positioning mode and a jump positioning mode. In this way, positioning can be achieved by loading only a single segment of the map corresponding to the submap without having to load the entire map.

[0077] When the system is initialized, the positioning function is turned on by default. At this time, the system is in the positioning state, and the positioning object is at the starting point of positioning. At this time, step S12 may include: determining the first position of the starting point (for example, the absolute coordinates in the absolute coordinate system) based on the data of the external sensor, searching for the corresponding subgraph within the predetermined range of the first position of the starting point based on the global subgraph segment endpoint library, importing the key frame database of the corresponding subgraph, using the DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculating the comprehensive similarity, and using the subgraph with the highest comprehensive similarity as the optimal subgraph of the starting point (i.e., the starting point graph), and loading the visual map corresponding to the optimal subgraph of the starting point.

[0078] In some embodiments, step S12 may further include switching the optimal sub-map upon detecting that the positioning function has switched from an off state to an on state. Specifically, when the positioning function is off, the current sub-map of the system is closed; when the positioning function is on, the optimal sub-map required for system positioning is re-determined using one of a continuous positioning mode and a jump positioning mode.

[0079] In some embodiments, Figure 2 Figure 2 shows a flow chart of determining the optimal subgraph after the system is initialized. Figure 2 As shown, after the system initialization is completed, step S12 may include:

[0080] Step 122: Update the number of sub-graphs with consecutive positioning failures based on the positioning status of the previous sub-graph;

[0081] In some implementations, the number of consecutive positioning failure subgraphs may be updated according to the following formula (1):

[0082]

[0083] Among them, n 连续定位失败子图 Indicates the number of consecutive subgraphs that failed to locate.

[0084] Step 124: If the number of subgraphs that have failed to be positioned continuously is less than or equal to the first number threshold, select a continuous positioning mode to determine the optimal subgraph. The continuous positioning mode includes: determining the next subgraph of the current subgraph as the optimal subgraph;

[0085] Step 126: If the number of subgraphs that have failed to be located continuously is greater than the first threshold, a jump positioning mode is selected to determine the optimal subgraph. The jump positioning mode includes: determining the set of similar frames in each subgraph using a DBoW-based bag-of-words model based on the global key frame database, calculating the comprehensive similarity, determining the subgraph with the highest comprehensive similarity as the optimal subgraph, and loading its visual subgraph.

[0086] As can be seen from the above, the embodiment of the present disclosure uses only a single sub-map for positioning, and adopts a sub-map switching method with continuous positioning as the main method and jump positioning as the auxiliary method when determining the sub-map to be loaded. The sub-map required for positioning is determined in real time according to the trajectory position and positioning situation, and it can not only reduce the memory usage of the map and the computing power consumption of positioning, but also improve the robustness and continuity of the positioning function.

[0087] Specifically, the embodiment of the present disclosure only needs to import a single sub-map during the positioning process, and there is no need to import the entire map, which reduces the space and computing power occupied by the visual map when it is reused; and, when the sub-map is successfully positioned (i.e., the positioning is smooth), the embodiment of the present disclosure uses a continuous positioning mode to sequentially import the sub-maps, avoiding unnecessary calculation processes and achieving a more reasonable allocation of resources; when the continuous positioning of multiple sub-maps fails, the embodiment of the present disclosure turns on the jump positioning mode and performs positioning in the global key frame database, which can achieve more robust and continuous positioning.

[0088] Step S14, performing positioning by matching images from separate cameras and cross-matching images from multiple cameras, using a map frame database corresponding to the optimal sub-image and the first pose obtained based on data from an external sensor, to obtain a second pose;

[0089] In some embodiments, in step S14, a positioning algorithm based on a multi-camera adaptive matching method of a rigid body combination method can be used to perform branch camera image matching and multi-camera image cross-matching, and positioning is performed using the map frame database corresponding to the optimal sub-image and the first pose obtained based on data from an external sensor to obtain a second pose.

[0090] Figure 3 FIG. 1 shows an exemplary flow chart of positioning. Figure 3 As shown, step S14 may specifically include:

[0091] Step S142, determining the initial pose of the current anchor point based on the positioning of the previous anchor point and the relationship between the current anchor point and the initial frame of the sub-image of the optimal sub-image;

[0092] In some implementations, if the current anchor point is the initial frame of the current sub-image or the previous anchor point positioning fails, the initial pose of the current anchor point can be determined as follows:

[0093] First, based on the DBoW-based bag-of-words model, the map anchor point with the highest comprehensive similarity to the multi-camera image of the current anchor point is searched in the key frame database of the optimal subgraph determined in step S12, and the pose of the map anchor point is used as the initial pose of the current anchor point.

[0094] Secondly, based on the initial position of the current anchor point, the map points corresponding to each camera in the map anchor point are projected onto the image of the camera at the same location as the current anchor point to obtain the projection points of each camera. Feature matching is performed within a certain range of the projection points of each camera to obtain the matching relationship between the feature points of the current anchor point and the map points of the map anchor point. The matching relationship obtained through the projection points of each camera is the feature matching relationship of the corresponding camera relative to the anchor point coordinate system;

[0095] Finally, the initial pose T of the current anchor point is estimated by combining the external parameters and feature matching relationships of each camera relative to the anchor point coordinate system. fw For example, the initial pose T of the current anchor point can be estimated according to the following formula (2): fw .

[0096]

[0097] in, represents the inverse projection process of the i-th camera model, Represents the external parameters of the i-th camera relative to the anchor coordinate system, is the jth map point corresponding to the i-th camera of the map anchor point, express Projection point at the current anchor point Matching feature points, λ i,j express The depth on the back-projected ray vector.

[0098] In some implementations, methods for solving the absolute pose of a rigidly coupled multi-camera platform may include, but are not limited to, a generalized P3P method and a generalized PNP method.

[0099] In some embodiments, if the previous anchor point is successfully located, the initial pose of the current anchor point can be determined by estimating the initial pose T of the current anchor point based on the pose of the previous anchor point and the first pose determined based on the data of the external sensor.fw For example, the initial pose T of the current anchor point can be estimated according to the following formula (3): fw .

[0100]

[0101] in, The anchor point represents the pose of the previous anchor point. and T bw Represent the first pose corresponding to the previous anchor point and the current anchor point respectively.

[0102] Step S144: Optimize the initial pose of the current anchor point for the first time.

[0103] First, the feature point matching relationship of the valid camera set is determined based on the initial pose of the current anchor point and the map keyframe set:

[0104] Specifically, based on the initial pose of the current anchor point and the pose of the map key frame set in the map frame database, a map key frame set with a Euclidean distance less than a predetermined threshold can be selected as a candidate key frame set. The candidate key frame set is processed frame by frame. The map points of the candidate key frame are projected onto the camera image corresponding to the current anchor point, and feature matching is performed within a certain range of the projection point. The number of feature point pairs successfully matched by each camera is counted {n i ,1≤i≤k}, calculate the sum of the number of successfully matched feature point pairs According to the distribution of feature points of each camera, the number of feature points is greater than the first threshold value T n At the same time, the effective proportion of feature points is less than the second quantity threshold T r The valid cameras form a valid camera set.

[0105] Secondly, based on the number of cameras included in the valid camera set, it is decided whether to continue the current positioning process. Specifically, if the number of cameras included in the valid camera set is less than a preset third number threshold T cn , the positioning fails, the current positioning process can be ended and the process goes directly to step S148; if the number of cameras included in the valid camera set is greater than or equal to the preset third number threshold T cn , you can continue the current positioning process.

[0106] For example, the effective camera set can be expressed as the following formula (4):

[0107]

[0108] Finally, based on the first pose determined by the external sensor data, the external parameters of each camera, and the matching relationship between the feature points of the valid camera set, an optimization problem is constructed to perform the first optimization on the initial pose of the current anchor point.

[0109] Exemplarily, the optimization problem can be formulated as the following formula (5), by which the initial pose of the current anchor point is optimized for the first time, thereby obtaining the first optimized pose of the current anchor point.

[0110]

[0111] in, represents the projection process of the i-th camera model, Represents the external parameters of the i-th camera relative to the anchor coordinate system, is the jth map point corresponding to the i-th camera of the map anchor point, express Matching feature points at the projection point of the current anchor point. i ,1≤i≤k} represents the number of successfully matched feature point pairs of the i-th camera.

[0112] In a specific application, the first quantity threshold T n , the second quantity threshold T r and the third quantity threshold T cn The value or value range can be selected based on the actual situation.

[0113] For example, when the number of multi-channel cameras is 4, the first number threshold T n The value range of can be 50 to 100, and the first quantity threshold T can be appropriately reduced in the texture missing area. n , the texture repetitive area can appropriately increase the first quantity threshold T n .

[0114] For example, when the number of multi-channel cameras is 4, the second number threshold T r The value range of can be 0.6~0.8. The second quantity threshold T can be appropriately increased in large-angle turning areas. r , the texture repetition area can appropriately reduce the second number threshold T r .

[0115] For example, when the number of multi-channel cameras is 4, the third number threshold T cn The value of can be 2. If the number of multi-channel cameras increases, the third number threshold T can be appropriately increased. cn The value of .

[0116] Step S146, performing secondary optimization of the current anchor point pose by multi-camera feature cross matching based on the first optimized pose of the current anchor point to determine the positioning pose of the current anchor point;

[0117] First, based on the first optimized pose of the current anchor point, perform multi-camera cross-matching on the map points of the map anchor point and the feature points of the current anchor point to obtain the feature point matching relationship of the multi-camera cross-matching. Specifically, the map points of the i-th camera of the map anchor point can be projected onto the i+1-th camera of the current anchor point, and the map points of the k-th camera of the map anchor point can be projected onto the 1st camera of the current anchor point, and feature matching can be performed within the projected point area.

[0118] Then, based on the first pose determined by the external sensor data, the external parameters of the branch camera and the feature point matching relationship of the multi-camera cross-matching, an optimization problem is constructed, and the first optimized pose of the current anchor point is optimized again to obtain the optimal pose of the current anchor point, which is used as the positioning pose of the current anchor point.

[0119] Exemplarily, the optimization problem of quadratic optimization can be constructed according to the following formula (6), that is, the quadratic optimization of the current anchor point pose can be completed by formula (6).

[0120]

[0121] in, Represents the projection process of the map point corresponding to the i-th camera of the map anchor point to the i+1-th camera model of the current anchor point (when i=k, i+1=0), Represents the external parameters of the i-th camera relative to the anchor coordinate system, is the jth map point corresponding to the i-th camera of the map anchor point, express The matching feature point of the i+1th camera projection point at the current anchor point.

[0122] Step S148: Update the visually corrected odometer according to the positioning pose of the current anchor point and the first pose obtained based on the data of the external sensor. The updated visually corrected odometer pose is the second pose of the positioning object.

[0123] Specifically, if the current anchor point is successfully located, the visually corrected odometry is updated based on the current anchor point's location pose and the first pose determined by the external sensor data. If the current anchor point is not located, the visually corrected odometry can be directly updated based on the first pose determined by the external sensor data.

[0124] As can be seen from the above, the embodiments of the present disclosure utilize the strategies of branch camera matching and multi-channel camera cross-matching to expand the multi-camera combination method, determine the positioning strategy and tracking method according to the distribution of branch camera feature points, realize adaptive multi-camera repositioning and tracking, improve the robustness of the positioning system of the positioning object in scenes with different information richness and the success rate in high-difficulty positioning scenes, and solve the problems of limited field of view of single-channel cameras, single feature matching strategy, poor positioning robustness and low accuracy in complex running trajectories and changing scenes.

[0125] Specifically, due to the large field of view of multiple cameras, they can fully capture scene information. In applications where positioning is challenging, such as in texture-deficient areas or when the robot is making wide turns, the disclosed embodiments employ multi-camera positioning to significantly improve positioning success rate, accuracy, and robustness. Secondly, the disclosed embodiments employ a multi-camera adaptive matching approach, utilizing a strategy of split-channel camera matching and multi-channel camera cross-matching to fully utilize scene information and significantly reduce the probability of mismatches in areas with repeated textures.

[0126] Step S16: according to the second posture, the positioning function is turned on or off by evaluating the track distance information and / or the image texture information, that is, by evaluating the track distance information and / or the image texture information, it is determined whether to pause or continue positioning.

[0127] In some implementations, whether to enable the positioning function of the positioning object can be determined by combining the current positioning state with the distance texture consistency.

[0128] In some embodiments, if the system is currently in a positioning state, step S16 may include: determining the position of the second posture determined in step S14 and a preset first distance threshold (ie, threshold T below). curd ), determines whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the end frame of the sub-graph.

[0129] Specifically, if the system is in the positioning state, the visually corrected odometer obtained in step S14 can be used to calculate the trajectory distance d between the current anchor point and the sub-image end frame. 当前子图 , when d 当前子图 Less than the threshold T curd You can turn off the positioning function, that is, pause positioning; when d 当前子图 Greater than or equal to the threshold T curd When the positioning function is turned on, the positioning function can be kept on, that is, positioning can be continued.

[0130] For example, the threshold T curd The value range of can be set to 10 meters to 20 meters. When the movement speed increases, the threshold T can be appropriately increased. curd When the camera shooting frequency increases, the threshold T can be appropriately lowered.curd It should be noted that the threshold T curd The adjustment method and value range of the threshold T are not limited to this example. In specific applications, the threshold T can be flexibly adjusted as needed. curd The real-time value or value range of .

[0131] In some embodiments, if the system is currently in a non-positioning state, step S16 may include: determining the position of the system according to the second position obtained in step S14, the first position obtained based on the data of the external sensor, and a preset second distance threshold (i.e., the threshold T below). nextd ) and the fourth quantity threshold (ie, the threshold T below h ), by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image, it is determined whether to enable the positioning function.

[0132] Specifically, if the system is in a non-positioning state, the specific implementation process of step S16 may include the following steps:

[0133] Step a1: Calculate the gray level co-occurrence matrix information entropy of each camera image corresponding to the current anchor point based on the following formula (7). The gray level co-occurrence matrix information entropy can be used to characterize the richness of texture information:

[0134]

[0135] Among them, h i is the gray-level co-occurrence matrix information entropy of the i-th camera image of the current anchor point, G i is the normalized gray-level co-occurrence matrix of the i-th camera image of the current anchor point, and l is the grayscale level.

[0136] Step b2: Determine the pose of the current anchor point using the visually corrected odometer obtained by the system in the positioning state (i.e., the second pose obtained in step S14) and the first pose obtained based on the data of the external sensor in the current non-positioning state, and calculate the trajectory distance d from the current anchor point to the starting frame of the next sub-graph based on the pose of the current anchor point. 下一子图 ;

[0137] Step b3: Determine whether the current state is one of the following two: If yes, turn on the positioning function and switch from the non-positioning state to the positioning state, that is, the system starts positioning; otherwise, the positioning function can be kept off, that is, the system remains in the non-positioning state:

[0138] 1)d 下一子图 Less than the threshold T nextd ;

[0139] 2) The effective camera set {C i |i∈{i|hi >T h and 1≤i≤k}} the number of cameras is greater than the threshold T h .

[0140] For example, the threshold T nextd The value range of can be set to 10 meters to 20 meters. When the movement speed of the positioning object (for example, a robot or a vehicle, etc.) increases, the threshold T can be appropriately increased. nextd .

[0141] Since the information entropy h i The value range is [0,2logl], so the threshold T h The selection of is related to the setting of gray level l. For example, when gray level l = 16, the threshold T h The value range of can be set to 0.8~1.5. When the gray level increases, the threshold T can be appropriately increased. h .

[0142] It should be noted that the threshold T nestd and threshold T h The adjustment method and value range of T are not limited to the above examples. In specific applications, the threshold T can be flexibly adjusted as needed. nestd and threshold T h The real-time value or value range of .

[0143] In this way, disabling positioning in areas without maps or textures reduces computing power consumption, which can avoid resource waste and reduce the probability of mismatches to a certain extent; turning on positioning in areas with map coverage and information-rich scenes is a more reasonable way to utilize resources, which can improve the accuracy, continuity, and success rate of positioning. As a result, segmented positioning based on distance-texture consistency judgment is achieved, which can fully control the timing of system positioning. This allows positioning to be turned off in areas with texture missing while maximizing positioning in areas with rich textures while considering the distribution range of the segmented sub-maps. This rationally allocates computing resources, fully explores the positioning potential of the scene, and achieves robust, continuous, and high-precision positioning with low computing power consumption, solving the problems of difficulty and low efficiency in positioning in areas with low scene information richness.

[0144] The above-mentioned method of the embodiment of the present disclosure controls the positioning timing for positioning and tracking based on distance-texture consistency judgment. Only a single sub-image is loaded during positioning. When determining the sub-image to be loaded, a sub-image switching method is adopted with continuous positioning as the main method and jump positioning as the auxiliary method. The required sub-image is determined in real time according to the trajectory and positioning conditions. During the positioning process, the positioning strategy is adaptively determined according to the distribution of feature points of the branch camera, thereby achieving robust, continuous, and high-precision positioning with low computing power consumption.

[0145] Figure 4 It is a schematic block diagram of the structure of a multi-camera segmented positioning device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0146] The device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by a corresponding module, and the device may include one or more of these modules. The module may be one or more hardware modules specifically configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for execution by a processor, or implemented by some combination thereof.

[0147] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 500 connects various circuits including one or more processors 600, memory 700, and / or hardware modules. Bus 500 can also connect various other circuits 800 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0148] Bus 500 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, only one connecting line is used in this figure, but this does not mean that there is only one bus or only one type of bus.

[0149] Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor performs the various methods and processes described above. For example, the method embodiments of the present disclosure can be implemented as a software program that is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0150] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device).

[0151] For the purposes of this specification, a "readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection having one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium can even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a memory.

[0152] It should be understood that various parts of the present disclosure can be implemented using hardware, software, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0153] Those skilled in the art will understand that all or part of the steps of the above-mentioned implementation method can be accomplished by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes one or a combination of the steps of the method implementation method.

[0154] Furthermore, the functional units in the various embodiments of the present disclosure may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as independent products, they may also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0155] Figure 4 FIG. 1 is a structural diagram of a multi-channel camera segmented positioning device according to an embodiment of the present disclosure. Figure 4 As shown, the multi-camera segmented positioning device 400 of the embodiment of the present disclosure may include:

[0156] A submap switching module 402 is configured to determine an optimal submap based on data from external sensors and prior visual map data;

[0157] The multi-camera relocalization and tracking module 404 is configured to obtain a second pose by performing image matching of separate cameras and cross-matching of images of multiple cameras, using a map frame database corresponding to the optimal sub-image and a first pose obtained based on data from external sensors;

[0158] The positioning segmentation module 406 is configured to enable or disable the positioning function by evaluating the trajectory distance information and / or the image texture information according to the second posture.

[0159] In some embodiments, the subgraph switching module 402 can be specifically used to determine the optimal subgraph required for system positioning based on the running trajectory and positioning conditions, using one of the continuous positioning mode and the jump positioning mode; the continuous positioning mode includes: determining the next subgraph of the current subgraph as the optimal subgraph; the jump positioning mode includes: based on the global key frame database in the prior visual map data, using the DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculate the comprehensive similarity, and determine the subgraph with the highest comprehensive similarity as the optimal subgraph.

[0160] In some embodiments, the subgraph switching module 402 can be specifically used to: determine the first position of the starting point based on data from an external sensor when the system is initialized, search for the corresponding subgraph within a predetermined range of the first position of the starting point based on a global subgraph segment endpoint library in the prior visual map data, import the key frame database of the corresponding subgraph, use a DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculate the comprehensive similarity, and use the subgraph with the highest comprehensive similarity as the optimal subgraph of the starting point.

[0161] In some implementations, the sub-map switching module 402 may be specifically configured to: when the positioning function is enabled, use one of the continuous positioning mode and the jump positioning mode to re-determine the optimal sub-map required for system positioning.

[0162] In some embodiments, the multi-camera relocalization and tracking module 404 can be specifically used to: use a positioning algorithm based on a multi-camera adaptive matching method of rigid body combination to perform branch camera image matching and multi-camera image cross-matching, and use the map frame database corresponding to the optimal sub-image and the first pose obtained based on data from external sensors for positioning, thereby obtaining a second pose.

[0163] In some embodiments, the multi-camera relocalization and tracking module 404 can be specifically used to: determine the initial pose of the current anchor point based on the positioning of the previous anchor point and the relationship between the current anchor point and the initial frame of the sub-image of the optimal sub-image; optimize the initial pose of the current anchor point through branch camera image matching to obtain the first optimized pose of the current anchor point; optimize the first optimized pose of the current anchor point through multi-camera image cross-matching to obtain the positioning pose of the current anchor point; and update the visually corrected odometer according to the positioning pose of the current anchor point and the first pose obtained based on data from external sensors, and the updated pose of the visually corrected odometer is the second pose.

[0164] In some embodiments, the positioning segmentation module 406 can be specifically used to: when the system is currently in a positioning state, determine whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the end frame of the sub-image based on the second posture and a pre-set first distance threshold.

[0165] In some implementations, the positioning segmentation module 406 may be specifically configured to disable the positioning function when the trajectory distance from the current anchor point to the end frame of the sub-image is less than a first distance threshold.

[0166] In some embodiments, the positioning segmentation module 406 can be specifically used to: when the system is currently in a non-positioning state, determine whether to turn on the positioning function based on the second posture, the first posture obtained based on data from an external sensor, and a pre-set second distance threshold and a fourth quantity threshold, by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image.

[0167] In some embodiments, the positioning segmentation module 406 can be specifically used to: turn on the positioning function in one of the following situations: 1) the trajectory distance from the current anchor point to the starting frame of the next sub-image is less than the second distance threshold; 2) the grayscale co-occurrence matrix information entropy of the camera image corresponding to the current anchor point is greater than the fourth quantity threshold, and the number of cameras in the valid camera set is greater than the fourth quantity threshold.

[0168] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executing the execution instructions stored in the memory, so that the processor or other hardware module performs the above-mentioned multi-camera segmented positioning method.

[0169] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned multi-channel camera segmented positioning method.

[0170] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are mutually inconsistent.

[0171] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0172] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A multi-channel camera segmented positioning method, characterized in that: include: Determine the optimal submap based on data from external sensors and prior visual map data; Through split-channel camera image matching and multi-channel camera image cross-matching, positioning is performed using the map frame database corresponding to the optimal sub-graph and the first pose obtained based on the data of the external sensor to obtain the second pose, including: using a positioning algorithm of a multi-camera adaptive matching method based on a rigid body combination method to perform split-channel camera image matching and multi-channel camera image cross-matching, positioning is performed using the map frame database corresponding to the optimal sub-graph and the first pose obtained based on the data of the external sensor, thereby obtaining the second pose, wherein the initial pose of the current anchor point is determined according to the positioning status of the previous anchor point and the relationship between the current anchor point and the initial frame of the sub-graph of the optimal sub-graph; the initial pose of the current anchor point is optimized through split-channel camera image matching to obtain the first optimized pose of the current anchor point; the first optimized pose of the current anchor point is optimized through multi-channel camera image cross-matching to obtain the positioning pose of the current anchor point; the visually corrected odometer is updated according to the positioning pose of the current anchor point and the first pose obtained based on the data of the external sensor, and the updated visually corrected odometer pose is the second pose; Based on the second pose, the localization function is turned on or off by evaluating the trajectory distance information and / or the image texture information.

2. The multi-camera segmented positioning method according to claim 1, characterized in that: The determining of the optimal submap based on the data of the external sensor and the prior visual map data includes: According to the running trajectory and positioning situation, one of the continuous positioning mode and the jump positioning mode is used to determine the optimal subgraph required for system positioning; The continuous positioning mode includes: determining the next sub-graph of the current sub-graph as the optimal sub-graph; The jump positioning mode includes: determining the similar frame set in each subgraph using a DBoW-based bag-of-words model based on a global keyframe database in the prior visual map data, calculating the comprehensive similarity, and determining the subgraph with the highest comprehensive similarity as the optimal subgraph.

3. The multi-camera segmented positioning method according to claim 2, characterized in that: According to the running trajectory and positioning situation, one of the continuous positioning mode and the jump positioning mode is used to determine the optimal subgraph required for system positioning, including: According to the positioning status of the previous sub-graph, the number of sub-graphs that failed to be positioned continuously is updated; If the number of the consecutive positioning failure subgraphs is less than or equal to a first number threshold, selecting a continuous positioning mode to determine an optimal subgraph; If the number of the consecutive sub-graphs that fail to be positioned is greater than a first number threshold, a jump positioning mode is selected to determine the optimal sub-graph.

4. The multi-channel camera segmented positioning method according to claim 1 or 2, characterized in that: The determining of the optimal submap based on the data of the external sensor and the prior visual map data further includes: When the system is initialized, the first position of the starting point is determined based on the data of the external sensor. The corresponding subgraph is searched within the predetermined range of the first position of the starting point based on the global subgraph segment endpoint library in the prior visual map data. The key frame database of the corresponding subgraph is imported, and the similar frame set in each subgraph is determined using the DBoW-based bag-of-words model. The comprehensive similarity is calculated, and the subgraph with the highest comprehensive similarity is used as the optimal subgraph of the starting point.

5. The multi-camera segmented positioning method according to claim 1 or 2, characterized in that: The determining of the optimal submap based on the data of the external sensor and the prior visual map data further includes: when the positioning function is turned on, using one of the continuous positioning mode and the jump positioning mode to re-determine the optimal submap required for system positioning.

6. The multi-camera segmented positioning method according to claim 1, characterized in that: The method of turning on or off the positioning function by evaluating the trajectory distance information and / or image texture information according to the second posture includes: when the system is currently in a positioning state, determining whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the end frame of the sub-image according to the second posture and a preset first distance threshold.

7. The multi-camera segmented positioning method according to claim 6, characterized in that: The method of turning on or off the positioning function by evaluating the trajectory distance information and / or the image texture information according to the second posture includes: turning off the positioning function when the trajectory distance from the current anchor point to the end frame of the sub-image is less than the first distance threshold.

8. The multi-camera segmented positioning method according to claim 1, characterized in that: The method of turning on or off the positioning function by evaluating trajectory distance information and / or image texture information according to the second posture includes: when the system is currently in a non-positioning state, determining whether to turn on the positioning function by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image according to the second posture, the first posture obtained based on data from an external sensor, and a pre-set second distance threshold and fourth quantity threshold.

9. The multi-camera segmented positioning method according to claim 8, characterized in that: The method of enabling or disabling the positioning function by evaluating the trajectory distance information and / or the image texture information according to the second posture includes: Enable the positioning function in one of the following situations: The trajectory distance from the current anchor point to the starting point frame of the next sub-image is less than the second distance threshold; The number of cameras in the valid camera set whose gray level co-occurrence matrix information entropy of the camera image corresponding to the current anchor point is greater than the fourth quantity threshold is greater than the fourth quantity threshold.

10. A multi-channel camera segmented positioning device, characterized in that: include: A submap switching module is used to determine the optimal submap based on the data from external sensors and the prior visual map data; A multi-camera relocalization and tracking module is configured to perform branch camera image matching and multi-channel camera image cross-matching, and to perform positioning using a map frame database corresponding to the optimal subgraph and a first pose obtained based on data from an external sensor, so as to obtain a second pose. Specifically, the module is configured to perform branch camera image matching and multi-channel camera image cross-matching using a map frame database corresponding to the optimal subgraph and a first pose obtained based on data from an external sensor, so as to obtain a second pose. The module determines the initial pose of the current anchor point based on the positioning of the previous anchor point and the relationship between the current anchor point and the initial frame of the subgraph of the optimal subgraph. Optimize the initial pose of the current anchor point through split camera image matching to obtain the first optimized pose of the current anchor point; The first optimized pose of the current anchor point is optimized by cross-matching images from multiple cameras to obtain the positioning pose of the current anchor point; Updating the visually corrected odometry according to the positioning pose of the current anchor point and the first pose obtained based on data from the external sensor, where the updated visually corrected odometry pose is the second pose; The positioning segmentation module is used to enable or disable the positioning function by evaluating the trajectory distance information and / or the image texture information according to the second posture.

11. The multi-channel camera segmented positioning device according to claim 10, characterized in that: The sub-map switching module is specifically used to determine the optimal sub-map required for system positioning by adopting one of the continuous positioning mode and the jump positioning mode according to the running trajectory and positioning situation; The continuous positioning mode includes: determining the next sub-graph of the current sub-graph as the optimal sub-graph; The jump positioning mode includes: determining the similar frame set in each subgraph using a DBoW-based bag-of-words model based on a global keyframe database in the prior visual map data, calculating the comprehensive similarity, and determining the subgraph with the highest comprehensive similarity as the optimal subgraph.

12. The multi-channel camera segmented positioning device according to claim 10, characterized in that: The subgraph switching module is specifically used to determine the first position of the starting point according to the data of the external sensor during system initialization, search for the corresponding subgraph within a predetermined range of the first position of the starting point according to the global subgraph segment endpoint library in the prior visual map data, import the key frame database of the corresponding subgraph, use the DBoW-based bag-of-words model to determine the set of similar frames in each subgraph, calculate the comprehensive similarity, and use the subgraph with the highest comprehensive similarity as the optimal subgraph of the starting point.

13. The multi-channel camera segmented positioning device according to claim 10, characterized in that: The sub-image switching module is specifically configured to, when the positioning function is enabled, adopt one of the continuous positioning mode and the jump positioning mode to re-determine the optimal sub-image required for system positioning.

14. The multi-channel camera segmented positioning device according to claim 10, characterized in that: The positioning segmentation module is specifically used to: when the system is currently in a positioning state, determine whether to turn off the positioning function by calculating the trajectory distance from the current anchor point to the sub-image end frame according to the second posture and a preset first distance threshold.

15. The multi-channel camera segmented positioning device according to claim 14, characterized in that: The positioning segmentation module is specifically configured to disable the positioning function when the trajectory distance from the current anchor point to the end frame of the sub-image is less than the first distance threshold.

16. The multi-channel camera segmented positioning device according to claim 10, characterized in that: The positioning segmentation module is specifically used to: when the system is currently in a non-positioning state, determine whether to turn on the positioning function by evaluating the richness of texture information and calculating the trajectory distance from the current anchor point to the starting frame of the next sub-image based on the second posture, the first posture obtained based on data from the external sensor, and the pre-set second distance threshold and fourth quantity threshold.

17. The multi-channel camera segmented positioning device according to claim 16, characterized in that: The positioning segmentation module is specifically used to: Enable the positioning function in one of the following situations: The trajectory distance from the current anchor point to the starting point frame of the next sub-image is less than the second distance threshold; The number of cameras in the valid camera set whose gray level co-occurrence matrix information entropy of the camera image corresponding to the current anchor point is greater than the fourth quantity threshold is greater than the fourth quantity threshold.

18. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the multi-channel camera segmented positioning method according to any one of claims 1 to 9.

19. A readable storage medium, characterized in that The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the multi-channel camera segmented positioning method according to any one of claims 1 to 9.

Images