A visual positioning method and system for wide-area edge-cloud collaborative computing

Abstract

The application discloses a kind of wide-area-oriented end edge cloud collaborative computing visual positioning method and system.For the mobile terminal with binocular camera, the computing framework of end edge cloud is used, and the online map established in advance is used to realize the real-time positioning and mapping based on binocular camera.In the end edge cloud framework, the end side undertakes the work of visual odometry and global map optimization using binocular camera.The end side transmits information to the edge computing unit, while the edge computing unit performs image retrieval and global positioning work.Finally, the positioning result calculated by the end edge collaborative computing is sent to the cloud for real-time visualization.The method improves the calculation efficiency of real-time positioning, and also supports the deployment mode of multiple mobile terminals connected to the edge cloud side, estimates and optimizes the positioning information of each terminal through data exchange, to obtain higher management efficiency.

Description

Technical Field

[0001] This invention relates to visual positioning methods in the field of intelligent computing, and more particularly to a visual positioning method and apparatus for wide-area edge-cloud collaborative computing. Background Technology

[0002] With the rapid development of advanced wireless communication technologies such as 5G and edge-cloud wide-area collaborative computing, intelligent transportation based on vehicle-road-cloud collaborative perception computing has become possible.

[0003] Currently, mainstream visual localization algorithms (such as the mainstream SLAM algorithm) concentrate all computational modules within mobile devices (such as vehicles). However, mobile computing units often have relatively limited computing power due to lightweight requirements, resulting in low efficiency, poor accuracy, and slow updates. In contrast, distributed collaborative visual localization methods distribute the various computational modules across different computing units on the device, edge, and cloud. This allows some computational tasks that would otherwise be performed on the mobile device to be moved to other computing units, improving the overall efficiency of the localization algorithm.

[0004] However, with the expansion of the operating environment, single-machine SLAM becomes inefficient, and the estimation time increases, often failing to achieve satisfactory results. Currently, some researchers in academia have proposed vision-based cooperative localization methods. Patrik Schmuck's team proposed a multi-mobile device-based cooperative SLAM algorithm framework—CCM-SLAM—in 2018. This framework consists of multiple mobile devices and a server. The mobile devices are mainly responsible for visual odometry, while the server receives observations from all mobile devices, performs image detection, global localization, and global optimization.

[0005] In 2020, Ali J.'s team proposed the Edge-SLAM algorithm, which is based on the mainstream ORB-SLAM. To address the limited computing and memory resources of mobile devices, Edge-SLAM only retains visual odometry and local loop closure detection and optimization tasks on the mobile device, while the remaining global computing tasks are performed on the edge computing device. However, although the above method distributes multiple computing modules across different computing units, the problems encountered by ordinary SLAM at the algorithm level, such as cumulative error, still appear in these methods. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a visual positioning method and apparatus for wide-area edge-cloud collaborative computing.

[0007] To achieve the above objectives, this invention provides a visual positioning method for wide-area edge-cloud collaborative computing, which specifically includes the following steps:

[0008] A first aspect of this invention provides a visual positioning method for wide-area edge-cloud collaborative computing, the method comprising:

[0009] S1, Reconstruct the global map of the scene and load the global map reconstruction onto the edge;

[0010] S2 acquires images by deploying binocular cameras at each end, and then performs optical flow tracing after aligning the timestamps of the images acquired by the binocular cameras using the visual odometry method to generate feature points and observation information.

[0011] S3. Based on the feature points and observation information obtained in step S2, estimate the pose of each frame on the end side, and perform local pose optimization and feature point optimization to obtain the initial localization result of the key frame.

[0012] S4, send the keyframe initial localization result and feature point information obtained in step S3 to the side;

[0013] S5. After receiving the initial positioning result of the key frame and the feature point information, the edge side performs global positioning to obtain positioning information and sends it back to the corresponding end side. The end side aligns the corresponding odometer coordinate system with the global map obtained in step S1 according to the positioning information, and completes the collaborative initialization of the end side odometer and the edge side global map.

[0014] S6. After the terminal side completes the initialization, it sends the key frame initial localization result and feature point information obtained in step S3 to the edge side. The edge side performs global localization based on the received key frame initial localization result, feature point information and global map, and feeds back the global localization result to the corresponding terminal side.

[0015] S7, after receiving the global positioning information, the edge performs global graph optimization;

[0016] S8 sends the corrected data obtained from global graph optimization to the edge for global positioning, and the edge transmits the global positioning information to the cloud for display.

[0017] A second aspect of this invention provides a visual positioning system for wide-area edge-cloud collaborative computing, used to implement the aforementioned visual positioning method for wide-area edge-cloud collaborative computing, the system comprising:

[0018] On the edge, it is used to perform optical flow tracing after aligning the timestamps of the images acquired by the binocular camera, generate feature points and observation information, estimate the pose of each frame on the edge using the feature points and observation information, and perform local pose optimization and feature point optimization to obtain the initial localization result of the key frame; it is also used to optimize the global map of the global localization information.

[0019] On the side, global localization is performed based on the initial localization results of keyframes, feature point information, and the global map;

[0020] The cloud is used to display the optimized global positioning results.

[0021] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described visual positioning method for wide-area edge-cloud collaborative computing.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described visual positioning method for wide-area edge-cloud collaborative computing.

[0023] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a visual positioning method for wide-area edge-cloud collaborative computing. It utilizes a mobile terminal equipped with a binocular camera, an edge-cloud computing framework, and a pre-established online map to achieve real-time localization and mapping (SLAM) based on the binocular camera. In the edge-cloud collaborative computing framework, the mobile terminal (i.e., the edge side) is responsible for visual odometry and global map optimization using the binocular camera. The mobile terminal transmits information to the edge computing unit, which performs image retrieval and global positioning. Finally, the positioning result calculated by the edge-cloud collaboration is sent to the cloud for real-time visualization. This invention's method can achieve real-time positioning and final visualization, and the various tasks are distributed across different computing units in the edge-cloud, thereby improving the computational efficiency of real-time positioning. The edge-cloud collaborative computing framework of this invention also supports a deployment mode where the edge-cloud side interfaces with multiple mobile terminals. Through data exchange, it estimates and optimizes the positioning information of each terminal, thereby achieving higher management efficiency. Attached Figure Description

[0024] Figure 1 This is a simplified system flowchart of an embodiment of the present invention;

[0025] Figure 2 This is a detailed system flowchart of an embodiment of the present invention;

[0026] Figure 3This is a schematic diagram of multi-machine deployment according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the global positioning results in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the cloud display effect in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of a class of methods consistent with some aspects of the invention as detailed in the appended claims.

[0031] The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0032] like Figure 1 As shown, the visual localization method for wide-area edge-cloud collaborative computing proposed in this invention specifically includes the following steps:

[0033] S1 performs a global map reconstruction of the scene and loads the global map reconstruction into the edge unit.

[0034] Specifically, image data is collected using drones and ground vehicles. Ground and aerial maps are then constructed from the image data collected by the drones and ground vehicles, respectively. Based on the camera attitude, the coordinate system relationship between the ground and aerial maps is estimated, and map fusion is performed to generate a complete global map. When loading the global map at the edge, a bag-of-words model is used to generate and load a database. In subsequent steps, the camera pose data and camera image information are loaded using sequences from the database.

[0035] S2 acquires images using binocular cameras deployed at each end. The images are then time-stamped and optically traced using visual odometry to generate feature points and corresponding observation information.

[0036] Specifically, when the binocular cameras deployed at each end begin to acquire images, the binocular cameras send the image information to the corresponding ROS topic. The image information of the ROS topic is processed by visual odometry, and after the image information is timestamped, features are extracted by optical flow tracing to generate feature points and corresponding observation information.

[0037] S3. Estimate the pose of each frame on the end side using the feature points obtained in step S2 and the corresponding observation information, and perform local pose and feature point optimization to obtain the initial localization result of the key frame.

[0038] Specifically, in this example, PnP technology is used to estimate the pose of each frame on the edge, and this, combined with binocular information, is used to solve for the depth of feature points, and local pose and feature point optimization is performed. Local pose optimization is achieved through local nonlinear optimization, which uses a ten-frame sliding window as the basic unit and employs bundle adjustment to treat both pose and feature points as optimization variables, optimizing them simultaneously to minimize reprojection error.

[0039] Minimize reprojection error ξ * The calculation process is as follows:

[0040] The following formula can be used to establish a mutual conversion relationship between the pixel coordinates in the image coordinate system and the three-dimensional coordinates in the world coordinate system of the feature points obtained in step S2.

[0041]

[0042] In the formula, s i a is the scale factor. i b is the x-coordinate of the pixel. i X is the pixel ordinate, K is the intrinsic parameter, and X is the pixel coordinate. i Y is the x-coordinate in the world coordinate system. i Z is the ordinate in the world coordinate system. i The coordinates are in the world coordinate system.

[0043] s i u i =Kexp(ξ ∧ )P i

[0044] In the formula, u i Let P be the projected coordinates. i These are the feature points obtained in step S2 in world coordinates.

[0045]

[0046] Where ξ represents the pose (in Lie algebra form) of the binocular camera deployed on each end side.

[0047] S4. Send the keyframe initial localization result and feature point information obtained in step S3 to the side.

[0048] S5. After receiving the initial positioning result of the key frame and the feature point information, the edge side performs global positioning to obtain positioning information and sends it back to the corresponding end side. The end side aligns the corresponding odometer coordinate system with the global map obtained in step S1 according to the positioning information, and completes the collaborative initialization of the end side odometer and the edge side global map.

[0049] Furthermore, the specific implementation of S5 is as follows: The positioning system initialization method involves the terminal vehicle sending keyframes to the edge vehicle. Based on the keyframe information, the edge vehicle uses a bag-of-words model to detect images in the online map that are similar to the keyframes, performs global positioning, and obtains the transformation relationship between the terminal vehicle's coordinate system and the online map's coordinate system. The edge vehicle then transmits the calculated transformation relationship back to the terminal vehicle's odometer. The odometer adjusts its coordinate system to be consistent with the online map's coordinate system based on the transmitted transformation, and clears and resets other information, completing the collaborative initialization of the terminal vehicle's positioning system and the edge vehicle's online map.

[0050] S6. After the terminal side completes the initialization, it sends the keyframe initial localization result and feature point information obtained in step S3 to the edge side. The edge side performs global localization based on the received keyframe initial localization result, feature point information and global map, and feeds back the global localization result to the corresponding terminal side.

[0051] The initial localization results of the keyframe and the criteria for sending feature point information to the edge include: Criterion 1 is the average disparity with the previous keyframe. If the average disparity of the tracked feature points between the current frame and the latest keyframe exceeds a certain custom average disparity threshold, then the frame is considered a new keyframe. Criterion 2 is tracking quality. If the number of tracked features is lower than a certain custom feature number threshold, then the frame is considered a new keyframe. The purpose is to avoid the complete loss of tracked features.

[0052] The local keyframe object to be optimized on the device side contains the keyframe sequence number, odometry pose, optimized pose, and global localization constraint information.

[0053] The global localization process includes: The edge-side uses a bag-of-words model to detect images similar to the keyframes in the online map based on keyframe information, and performs global localization to obtain the relocalized pose of the keyframes in the online map coordinate system. In global localization, the system uses the BREIF descriptor to match the feature points of the keyframes with the online map, then calculates the relocalized pose using the PnP-RANSAC method, and finally filters out relocalization results with in-RNANSAC points below a certain threshold. Finally, the edge-side integrates the filtered relocalized poses and the sequence numbers of their respective keyframes and sends them to the graph optimization module on the client-side.

[0054] S7, after receiving the global positioning information, the image optimization module on the edge performs global image optimization and sends the corrected data to the visual odometry module.

[0055] Specifically, after receiving the global positioning information, the graph optimization module on the end side performs global graph optimization to obtain corrected data. The corrected data includes the calculated cumulative offset pose, which repositions the current vehicle to the original world coordinate system.

[0056] The global graph optimization process is as follows: a 6-DOF pose graph optimization is constructed, which only optimizes the trajectory. The edges between pose nodes are given initial values ​​by the motion estimates obtained after feature matching between two keyframes. In particular, once the initial estimation is completed, the positions of landmark points are no longer optimized; only the poses of all keyframes corresponding to the stereo cameras deployed on the edge are optimized.

[0057] The order edge residual r between frame i and frame j i,j for:

[0058]

[0059] Where, p i ,q i ,p j ,q j The positions and orientations of the i-th and j-th frames are respectively. and These represent the estimated relative position and orientation between the two frames, respectively.

[0060] The loopback residual of the i-th frame is

[0061]

[0062] in, These are the relocation position and direction of the i-th frame transmitted back from the side, respectively.

[0063] The overall objective function for all sequential edges and loop edges, i.e., the objective function for graph optimization, is:

[0064]

[0065] Where S is the set of sequential edges, L is the set of loop edges, and h(*) is the Huber function.

[0066] In S8, the visual odometry module on the edge sends the corrected real-time camera positioning result obtained from global map optimization to the edge for global positioning. The edge then transmits the global positioning information to the cloud for display.

[0067] Accordingly, this invention proposes a visual positioning system for wide-area edge-cloud collaborative computing, used to implement the aforementioned visual positioning method for wide-area edge-cloud collaborative computing. The system includes:

[0068] Several end-side units, edge-side units, and cloud-side units;

[0069] The edge unit includes a visual odometry module and a graph optimization module. The visual odometry module is used to perform optical flow tracing on the images acquired by the binocular camera after time-stamp alignment, generate feature points and observation information, estimate the pose of each frame on the edge using the feature points and observation information, and perform local pose optimization and feature point optimization to obtain the initial localization result of the keyframe. The graph optimization module is used to perform global graph optimization on the global localization information.

[0070] The side unit performs global positioning based on the initial positioning result of the keyframe, feature point information, and the global map;

[0071] The cloud unit is used to display the optimized global positioning results.

[0072] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0073] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0074] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the visual positioning method for wide-area edge-cloud collaborative computing as described above. Figure 6 The diagram shown is a hardware structure diagram of any device with data processing capabilities for the visual positioning method for wide-area edge-cloud collaborative computing provided in this embodiment of the invention, except... Figure 6 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0075] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the visual positioning method for wide-area edge-cloud collaborative computing as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0076] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0077] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A visual positioning method based on edge-cloud collaborative computing in wide area, characterized in that, The method comprises: S1, global map reconstruction is performed on a scene, and the global map reconstruction is loaded to a side; S2, images are collected by binocular cameras arranged on each end side, the images collected by the binocular cameras are timestamped and aligned by using a visual odometer calculation method, and then optical flow tracking is performed to generate feature points and observation information; S3, the pose of each frame of the end side is estimated based on the feature points and the observation information obtained in step S2, and local pose optimization and feature point optimization are performed to obtain a key frame initial positioning result; S4, the key frame initial positioning result and the feature point information obtained in step S3 are sent to the side; S5, after the side receives the key frame initial positioning result and the feature point information, global positioning is performed to obtain positioning information, and the positioning information is fed back to the corresponding end side, and the end side aligns the corresponding odometer coordinate system with the global map obtained in step S1 according to the positioning information, and the collaborative initialization of the odometer of each end side and the global map of the side is completed; S6, after the end side completes the initialization, the key frame initial positioning result and the feature point information obtained in step S3 are sent to the side, and the side performs global positioning according to the received key frame initial positioning result, the feature point information and the global map, and the global positioning result is fed back to the corresponding end side; S7, after the end side receives the global positioning information, global map optimization is performed; S8, the correction data obtained by the global map optimization is sent to the side for global positioning, and the global positioning information is transmitted to the cloud for display.

2. The visual positioning method of claim 1, wherein, The process of global map reconstruction in step S1 comprises: image data is collected by a drone and a ground vehicle, and then a ground map and an aerial map are obtained by respectively mapping the image data collected by the drone and the ground vehicle, the coordinate system relationship between the ground map and the aerial map is estimated according to the camera pose, map fusion is performed, and a complete global map is obtained.

3. The visual positioning method of claim 1, wherein, Step S2 specifically comprises: when the binocular cameras arranged on each end side start collecting images, the binocular cameras send image information to a corresponding ROS topic, the image information of the ROS topic is processed by a visual odometer calculation method, the image information is timestamped and aligned, and then feature extraction is performed by optical flow tracking to generate feature points and corresponding observation information.

4. The visual positioning method of claim 1, wherein, Step S3 comprises: the pose of each frame of the end side is estimated by using PnP technology, local nonlinear optimization is adopted, ten frame sliding windows are taken as basic units, the pose and the feature points are both taken as optimization variables by using a bundle adjustment method, and the optimization is simultaneously performed to minimize the re-projection error, and a key frame initial positioning result is obtained.

5. The visual positioning method of claim 1, wherein, In step S6, the process of sending the key frame initial positioning result and the feature point information obtained in step S3 to the side further comprises: screening the key frame initial positioning result and the feature point information according to a sending standard, and sending the screened key frame initial positioning result and the feature point information to the side; The sending standard comprises: the average disparity of the last key frame, if the average disparity of the feature points tracked between the current frame and the latest key frame exceeds a self-defined average disparity threshold, the frame is regarded as a new key frame; Tracking quality, if the number of tracked features is less than a custom feature quantity threshold, the frame is considered as a new key frame.

6. The visual positioning method of claim 1, wherein, In the step S6, the process of global positioning includes: The side detects the images similar to the key frame in the online map according to the key frame information by using the bag-of-words model, and performs global positioning to obtain the repositioning pose of the key frame in the online map coordinate system; in the global positioning, the feature points of the key frame are matched with the online map by using the BREIF descriptor, and then the PnP-RANSAC method is used to calculate the repositioning pose, and then the repositioning results with less than a threshold of RANSAC inliers are screened out; then, the side sends the repositioning pose obtained by screening and the sequence number of the key frame to the end side.

7. The visual positioning method of claim 1, wherein, The process of global graph optimization in the step S7 is specifically: The global graph optimization is only the 6-DOF pose graph optimization of the trajectory, and the edges between the pose nodes are given initial values by the motion estimation obtained after the feature matching between the two key frames; wherein the residual r of the ith frame and the jth frame is i,j is: where p i q i p j q j are the position and orientation of the i, j-th frame, and are the estimated relative position and orientation between the two frames; The loop residual of the i-th frame is: wherein, is the position and orientation of the relocation of the i-th frame of the edge-side backhaul; The overall objective function of all sequence edges and loop edges, that is, the objective function of graph optimization, is: Wherein, S is a set of sequential edges, L is a set of loop edges, and h(*) is a huber function.

8. A visual positioning system based on wide-area edge-cloud collaborative computing, configured to implement the visual positioning method based on wide-area edge-cloud collaborative computing according to any one of claims 1-7, characterized in that, The system comprises: The end side is configured to perform optical flow tracking on the images collected by the binocular camera after time stamp alignment, generate feature points and observation information, estimate the pose of each frame of the end side based on the feature points and the observation information, and perform local pose optimization and feature point optimization to obtain the initial positioning result of the key frame, and perform global graph optimization on the global positioning information. The side side is configured to perform global positioning based on the initial positioning result of the key frame, the feature point information and the global map. The cloud side is configured to display the global positioning result after optimization.

9. An electronic device comprising a memory and a processor, characterized in that The memory is coupled to the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the visual positioning method of the end-edge-cloud collaborative computing facing wide area according to any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the visual positioning method of the end-edge-cloud collaborative computing facing wide area according to any one of claims 1-7.

Images