A multi-threaded map updating method, storage medium, electronic device and program product

Abstract

The present application provides a multi-threaded map update method, storage medium, electronic device and program product. By caching data outside the process, the decoupling of the data relationship between each link in the map update method is achieved, and the conditional variables of the process are adjusted by updating the cached data to achieve the decoupling of the trigger logic between each link. Therefore, the present application can split the map update method into multiple independently running processes. In actual execution, each process can be executed in a different thread, thereby realizing multi-linear parallel update of the map to improve the execution speed of the map update. In particular, in order to eliminate the accumulated error in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be implemented simultaneously by calling the same functional module in parallel in different threads, so that the elimination of the accumulated error in the map does not affect the normal display and update of the map.

Description

Technical Field

[0001] The present application relates to the technical field of mobile equipment, and in particular to a multi-threaded map updating method, storage medium, electronic device and program product. Background Art

[0002] Autonomous devices are machines that can move on their own, such as embodied robots, cleaning robots, and electric vehicles. During their movement, autonomous devices typically use sensors integrated into them to perceive the external environment and determine their own motion, thereby building a perception map of the autonomous device and performing subsequent tasks.

[0003] Considering the computing power of mobile devices and the complexity of sensor data, how to improve the processing speed of map data is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a multi-threaded map update method, storage medium, electronic device and program product, which can decouple the various functions within the map update algorithm through out-of-process data caching, so that they can be executed simultaneously in different threads, realizing a multi-threaded map update to speed up the map update process.

[0005] In a first aspect, the present application provides a multi-threaded map update method, which is applied to a mobile device and includes:

[0006] The current frame at the current moment is obtained through the data acquisition process, and the current frame is copied to the corresponding data cache area based on the frame type of the current frame, wherein the current frame includes the point cloud data and pose data at the current moment, the frame type includes key frames and ordinary frames, and the data cache area includes ordinary frame cache partitions and key frame cache partitions.

[0007] In response to a first condition variable of a map update process satisfying a process wake-up condition, the map update process is woken up in a first thread, a current frame is called from a data cache area of ​​the map update process through the map update process, and a first navigation map of the mobile device is updated based on the current frame, wherein the first condition variable of the map update process reflects the data update status of a key frame cache partition and a common frame cache partition of the map update program.

[0008] In response to the conditional variable of the loop detection process satisfying the process wake-up condition, the loop detection process is woken up, and the current frame and multiple key frames are called from the key frame cache partition of the loop detection process through the loop detection process. The loop result of the current frame is determined based on the multiple key frames and the current frame and stored in the loop result cache partition of the data cache area. When the current frame is a loop frame, the loop result includes the current frame, the matching frame corresponding to the current frame, and the relative posture between the current frame and the matching frame. The conditional variable of the loop detection process reflects the data update status of the key frame cache partition of the loop detection process.

[0009] In response to the conditional variable of the loop closure optimization process satisfying the process wake-up condition, the loop closure optimization process is woken up, and the loop closure optimization results of multiple target frames between the current frame and the matching frame are determined based on the loop closure results by the loop closure optimization process and stored in the optimization record cache partition of the data cache area, wherein the conditional variable of the loop closure optimization process reflects the data update status of the loop closure result cache partition of the loop closure optimization process, and the loop closure optimization result includes the corrected pose data.

[0010] In response to the second condition variable of the map update process satisfying the process wake-up condition, the map update process is woken up in the second thread, and the second navigation map of the mobile device is determined based on the loop optimization results of multiple target frames through the map update process and replaces the first navigation map, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

[0011] In a second aspect, the present application provides an electronic device. The electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-threaded map update method described in the first aspect.

[0012] In a third aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the multi-threaded map update method as described in the first aspect when the computer program / instruction processor is executed.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program / instruction. When the computer program / instruction is executed by a processor, the multi-threaded map update method described in the first aspect is implemented.

[0014] The present application provides a multi-threaded map update method, storage medium, electronic device and program product. It relates to the field of robotics technology. The present application achieves the decoupling of the data relationship of each functional module in the map update method by caching data outside the process, and adjusts the conditional variables of the process by updating the cached data, thereby achieving the decoupling of the trigger logic between each functional module. Therefore, the present application can split the map update method into multiple independently running processes. In actual execution, each process can be executed in a different thread, thereby achieving multi-linear parallel update of the map to improve the execution speed of the map update. In particular, in order to eliminate the accumulated error in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be achieved simultaneously by calling the same functional module in parallel in different threads, so that the elimination of the accumulated error in the map does not affect the normal display and update of the map. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 This is a schematic diagram of the functional modules of a mobile device provided in some embodiments of the present application.

[0017] Figure 2 This is an exemplary flowchart of a map updating method provided in some embodiments of the present application.

[0018] Figure 3 This is an exemplary flowchart of a data caching process provided in some embodiments of the present application.

[0019] Figure 4 Schematic diagram of the loop correction process provided by some embodiments of the present application.

[0020] Figure 5 This is an exemplary flowchart of the first navigation map correction method provided in some embodiments of the present application.

[0021] Figure 6 This is a system module diagram of a multi-threaded map update system provided in some embodiments of the present application. DETAILED DESCRIPTION

[0022] Application Overview

[0023] A self-moving device may refer to an electronic device equipped with a mobile component. Among them, the self-moving device can achieve corresponding work tasks through its own movement. Its specific work tasks and movement methods can be determined based on the specific form of the self-moving device. For example, for cleaning robots (such as sweeping robots, lawn mowing robots, etc.), their movement methods generally rely on the moving devices on the chassis (such as tires, tracks, etc.), and their work tasks are generally performed based on their integrated cleaning devices (such as vacuum devices, grass cutting devices, etc.). For another example, for embodied robots (such as humanoid robots, bionic robots, etc.), their movement methods are generally also based on their integrated moving devices (such as humanoid mechanical feet, tires, etc.), and their work tasks are generally performed based on robot arms or other integrated devices.

[0024] In actual operations, autonomous mobile devices often integrate SLAM (Simultaneous Localization and Mapping) technology, considered a visual SLAM system. SLAM is a technology used in devices such as robots and autonomous vehicles to enable real-time localization and mapping in unknown environments. Its core task is to acquire environmental information through sensor data, use this information to determine the device's position and orientation within the environment, and gradually construct a map of the environment.

[0025] SLAM technology mainly relies on a variety of sensors to collect environmental information, including cameras, Lidar, Inertial Measurement Unit (IMU), etc. The data provided by these sensors can be used to estimate the device's pose (or transformation), that is, the device's position and posture in the world coordinate system, after being processed by algorithms.

[0026] However, in the visual SLAM system of mobile devices, the pose estimation is often a recursive process, that is, the pose of the current frame is solved by the pose of the previous frame. Therefore, the error is passed down from frame to frame, resulting in cumulative error.

[0027] In related technologies, loop closure detection can be used to overcome the aforementioned cumulative error. The basic principle of loop closure detection is to detect whether the current frame forms a loop with a previous keyframe. Specifically, when a keyframe is inserted, a loop closure detection thread is started to check whether the currently inserted keyframe forms a loop with the previous keyframe. If so, the map points and poses corresponding to these frames are retrieved in order based on the queried keyframe ID and the current frame ID, and the optimization function is executed.

[0028] For example, in a visual SLAM system, a camera acquires continuous image frames, calculates the camera's pose in the environment map in real time, selects keyframes at regular intervals, saves them to a keyframe queue, and stores the keyframe poses in the map information. In a loop detection task, for the current image frame Iq, a matching keyframe Id is found in an image queue Id = {In, n = 1, 2, ..., N} with N keyframes. Its relative rotation qR and translation t are calculated, and the actual pose of the current frame in the environment map is obtained after being compared with the saved pose of qId, thereby correcting the accumulated error calculated previously (such as adjusting the pose data to update the map).

[0029] The present application has found that in related technologies, the aforementioned process is often implemented in a mobile device based on the same program, that is, in a common map update algorithm, loop detection can be first performed based on the received sensor data, and loop optimization can be performed when the currently collected data frame is a loop frame, and the map data can be reconstructed based on the loop optimization results.

[0030] Furthermore, considering the computational complexity of loop detection, loop optimization, and map reconstruction, in actual applications, when executing the aforementioned processes, significant lag may occur during loop optimization, seriously affecting the operation of the mobile device.

[0031] Based on the above situation, in order to improve the operating efficiency of relevant algorithms, this application creatively decouples the internal data dependency and trigger relationship dependency of the aforementioned algorithms, and decomposes the entire optimization process into multiple processes running independently in different threads, thereby performing map updates in a multi-threaded form.

[0032] Specifically, the present application achieves the decoupling of the data relationships between various links in the map update method by caching data outside the process, and achieves the decoupling of the trigger logic between various links by adjusting the conditional variables of the process through the update of cached data. As a result, the present application can split the map update method into multiple independently running processes, and in actual execution, each process can be executed in a different thread, thereby achieving multi-linear parallel update of the map to improve the execution speed of the map update. In particular, in order to eliminate the accumulated errors in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be implemented by calling the same link in parallel on different threads, so that the elimination of the accumulated errors in the map does not affect the normal display of the map.

[0033] To further describe the self-moving device that realizes the aforementioned technology, the present application provides a functional module diagram of the self-moving device ( Figure 1 ).

[0034] In the present application, the self-moving device 100 may be an electronic device with a self-moving function. Specifically, the self-moving device may also be a semi-autonomous device or a fully autonomous device. For example, the self-moving device 100 may be an embodied robot, a cleaning robot, an electric vehicle, etc. In some embodiments, the self-moving device 100 may have an operating capability, and it is generally integrated with a corresponding operating device (such as a mowing rope, a robotic arm, a storage bin, etc.), which can be adjusted according to actual conditions, and this application will not elaborate on this.

[0035] The following, in conjunction with the accompanying drawings, provides a clear and complete description of the multi-threaded map update method and its related self-mobile device in the embodiments of this application. Obviously, the described embodiments are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of this application without inventive effort are also within the scope of protection of this application.

[0036] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] Exemplary multi-threaded map update method

[0040] To illustrate the multi-threaded map update method provided by this application, this application provides a module diagram of a mobile device ( Figure 1 ) and an exemplary flowchart of a multi-threaded map update method ( Figure 2 ). The following is combined Figure 1 、 Figure 2 The multi-threaded map updating method provided by this application is described in detail.

[0041] In accordance with the foregoing, this application Figure 1The self-mobile device 100 is generally integrated with the aforementioned visual SLAM system. That is, the self-mobile device 100 can perform real-time positioning and map construction in an unknown environment, thereby performing corresponding work tasks.

[0042] like Figure 1 As shown, the self-propelled device 100 may include a processor 110, a memory 120, and a sensor 130. The processor 110 may be communicatively connected to the memory 120 and the sensor 130. Furthermore, the self-propelled device 100 is often also equipped with a mobile device (e.g., tires, robot feet, etc.) for driving the self-propelled device 100 and an operating device (e.g., a robotic arm, a warehouse, etc.) for performing operations. The mobile device and operating device can be configured according to actual needs and are not described in detail here.

[0043] The processor 110 may refer to a collection of devices with computing capabilities in the mobile device 100. For example, the processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, a processor, or any other conventional processor.

[0044] The memory 120 may refer to a collection of devices with data storage capabilities in the mobile device 100. For example, the memory 120 may include a random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0045] The sensor 130 may be a collection of devices for acquiring sensor data at each moment. Among them, the sensor 130 may include at least an environmental sensor 131 for sensing the environment and a posture sensor 132 for sensing the posture of the self-mobile device 100. For example, the aforementioned environmental sensor 131 may generally include a radar sensor (such as a lidar, a millimeter-wave radar, etc.) and / or a camera sensor (such as a multi-eye camera, a depth-of-field camera, etc.). The environmental sensor 131 may sense the environment in which the self-mobile device 100 is located to determine its point cloud data. The aforementioned posture sensor 132 may generally be configured as an inertial sensor, thereby determining the inertial information at each moment to reflect the posture of the self-mobile device (such as performing time series integration).

[0046] In some embodiments, during the actual data collection process, the environmental sensor 131 and the posture sensor 132 can be synchronized in time and space, that is, point cloud data and posture data are collected at the same time and packaged into a data frame at the current moment.

[0047] During the actual map update process, the memory 120 may store instructions related to the map update algorithm. When the instructions are executed by the processor 110, the processor 110 can execute the multi-threaded map update method provided in this application.

[0048] Considering that the map update method provided herein is multi-threaded, the processor 110 has multi-threaded processing capabilities, and the memory 120 stores different steps of the map update method as different programs. Therefore, when executing the multi-threaded map update method provided herein, different programs can be called by different threads to generate corresponding processes, thereby implementing map updates in a multi-threaded parallel execution format.

[0049] Specifically, the programs in the aforementioned memory 120 may mainly include a data acquisition program, a map update program, a loop detection program, and a loop optimization program. Considering the current architecture of the visual SLAM system, the aforementioned data acquisition program may be the front-end tracking program of the visual SLAM system, while the map update program, the loop detection program, and the loop optimization program may be the back-end optimization program of the visual SLAM system.

[0050] In actual applications, the data collection program can provide collected data frames to each backend optimization program. That is, in a multi-threaded map update, the data collection program is configured as a producer, and the three backend optimization programs can be configured as consumers.

[0051] When the various programs (i.e., instruction sets) in the aforementioned memory 120 are called by the processor 110, they can be executed on different threads to form processes of multiple different threads. Among them, after the data acquisition program is called, a data acquisition process can be formed to determine the data frame at each moment (wherein the data frame at the current moment is recorded as the current frame). After the map update program is called, a map update process can be formed to update / determine the navigation map of the mobile device based on the data frame. After the loop detection program is called, a loop detection process can be formed to determine the loop result. After the loop optimization program is called, a loop optimization process can be formed to optimize the posture data of the relevant data frame based on the loop result. The specific processing procedures of each process can be found in the subsequent description and will not be repeated here.

[0052] In some embodiments, considering that in traditional methods, the aforementioned programs are generally integrated into a large program, to achieve relative independence of each link, this application creatively decouples data between the various programs by constructing a data cache area. Conditional variables are configured based on the data update status of the cache area, thereby decoupling the correlation between the various links. This allows the traditional single program to be converted into multiple independent programs, which can be executed in different threads, realizing multi-linear parallel map updates. In other words, each backend optimization program can be configured with a conditional variable that can monitor the data update status of the corresponding cache area and is used to adjust the triggering / blocking status of the corresponding process based on its specific value.

[0053] In some embodiments, taking into account the principle of loop optimization, the data cache area of ​​each process may include a key frame cache partition and a normal frame cache partition. In addition, taking into account the correlation between each link, the data cache area of ​​each process may also include a cache partition for the output results of the previous link. For example, the data cache partition of the aforementioned loop optimization program may include a loop result cache partition for storing the loop results in the loop detection program, and the aforementioned map update program may include an optimization record cache partition for storing the loop optimization results of the loop optimization program.

[0054] The aforementioned conditional variables can be configured based on the cache partition of the corresponding process. Specifically, the map update program can be configured with a first conditional variable and a second conditional variable, wherein the first conditional variable can be used to reflect the data update status of the key frame cache partition and the ordinary frame cache partition of the map update program. The second conditional variable can be used to reflect the data update status of the optimized record cache partition. The conditional variables of the loop detection program can be used to reflect the data update status of the key frame cache partition. The conditional variables of the loop optimization program can be used to reflect the data update status of the loop result cache partition. The specific triggering logic of each conditional variable can be found in the subsequent content and will not be repeated here.

[0055] In some embodiments, the aforementioned data acquisition program can also be configured with a data cache area, thereby implementing a two-layer cache. That is, when executing the multi-linear map update method of the present application, each process will only access its own data cache area, and will not access the data cache area of ​​the data acquisition program, thereby greatly reducing thread blocking.

[0056] Based on the above Figure 1 The relevant content of has described the hardware and software basis for executing the map updating method of this application. Figure 2 The actual execution process is described. Figure 2 The process P200 shown can be executed by the aforementioned self-mobile device.

[0057] like Figure 2 As shown, P200 may include the following steps:

[0058] S210 , obtaining a current frame at a current moment through a data acquisition process, and copying the current frame to a corresponding data buffer area based on a frame type of the current frame.

[0059] S220, in response to the first condition variable of the map update process satisfying the process wake-up condition, wake up the map update process in the first thread, call the current frame from the data cache area of ​​the map update process through the map update process, and update the first navigation map of the mobile device based on the current frame.

[0060] S230, in response to the condition variable of the loop detection process satisfying the process wake-up condition, wake up the loop detection process, call the current frame and multiple key frames from the key frame cache partition of the loop detection process through the loop detection process, determine the loop result of the current frame based on the multiple key frames and the current frame, and store it in the loop result cache partition of the data cache area.

[0061] S240. In response to the condition variable of the loop closure optimization process satisfying the process wake-up condition, wake up the loop closure optimization process, determine the loop closure optimization results of multiple target frames between the current frame and the matching frame based on the loop closure results through the loop closure optimization process, and store them in the optimization record cache partition of the data cache area.

[0062] S250: In response to the second condition variable of the map update process satisfying the process wake-up condition, wake up the map update process in the second thread, and determine a second navigation map of the mobile device based on loop optimization results of multiple target frames through the map update process and replace the first navigation map.

[0063] Based on the above Figure 2As can be seen, process P200 contains two relatively independent processing flows: the regular map update process formed by S210 and S220, and the loop optimization process formed by S210, S230-S250. Relatively independent means that the regular map update process and the loop optimization process can run in parallel on different threads. Specifically, the map update process involved in the regular map update process runs on the first thread, while the map update process involved in the loop optimization process runs on the second thread.

[0064] In S210, during the operation of the self-mobile device, the sensor integrated in the self-mobile device (such as the aforementioned sensor 130) may periodically collect sensor data and encapsulate the sensor data at each moment into a data frame. The data frame corresponding to the current moment may be referred to as the current frame. Based on the foregoing, it can be seen that each data frame may include point cloud data and pose data at the corresponding moment.

[0065] In some embodiments, the sensor may not be able to directly acquire point cloud data and pose data during actual application. Instead, the corresponding sensor data may be acquired and processed to determine the point cloud data and pose data. Optionally, the point cloud data and pose data determination algorithm may be integrated into the data acquisition process or configured as a separate process.

[0066] For example, the point cloud data determination algorithm may include a coordinate conversion algorithm for the point cloud data. That is, the sensor data acquired by the sensor is often represented as depth information based on the sensor coordinate system. Therefore, when determining the point cloud data, the depth data may be subjected to a coordinate conversion to determine the depth information in the mobile device coordinate system and serve as the point cloud data. Optionally, the aforementioned determination algorithm may also include other depth data-related algorithms (such as conventional algorithms such as filtering and smoothing).

[0067] For another example, the algorithm for determining posture data can include odometry. That is, the sensor data corresponding to the posture data is often configured as multi-axis inertial data (such as acceleration in three directions and angular velocity in three directions). When determining the posture data, the odometry method can be used to integrate the acquisition interval between two data frames to determine the posture change between the two data frames, and then combine the posture data at the previous moment to determine the posture data at the current moment.

[0068] In some embodiments, when caching the current frame, it can be stored in different regions based on the frame type. That is, the data cache area of ​​each process is configured with a normal frame cache partition and a key frame cache partition, so that the partitioned storage is based on the frame type of the current frame. In addition, it should be noted that to distinguish each data frame, the data frame can be configured with identification information (such as a global ID, timestamp, etc.). The subsequent data update determination can also be determined based on whether the identification information of the latest data in the corresponding queue of the data cache area has not been processed.

[0069] In addition, as mentioned above Figure 1 As shown, each process can be configured with a data cache area. Then, in the aforementioned S210, the current frame can be copied one by one to each data partition of the data cache area of ​​each process. For more information about frame type determination and partition storage, please refer to Figure 3 and its related descriptions.

[0070] In some embodiments, when executing the aforementioned S210 , when data is updated in the data cache area of ​​each process, the corresponding process can be triggered based on the data update situation.

[0071] As mentioned above, the first condition variable of the map update process can monitor the data update status of the key frame buffer partition and the common frame buffer partition of the map update program, and thus be configured as a valid value when the data is updated to control the map update process to wake up in the first thread to execute the above S220.

[0072] As previously mentioned, the map update process can have a first thread and a second thread. Map update processes in different threads can be invoked based on the same map update program, thereby generating different navigation maps for different scenarios. The map update process in the first thread is generally resident in the first thread to update the corresponding navigation map based on data frame updates. The navigation map determined by the map update process in the first thread can be referred to as the first navigation map.

[0073] In some embodiments, the specific update method of the map update process can be configured based on actual needs. Figure 1 Generally, the configuration is a grid map. In the map update process, the occupancy status of each grid in the grid map can be determined based on the point cloud data to update the grid map.

[0074] Taking into account that when navigating, a self-mobile device generally only involves two-dimensional information, the aforementioned grid map can be configured as a two-dimensional grid map. Therefore, when determining the grid map, the point cloud data can be first filtered and projected based on the self-mobile device, retaining only the point cloud data related to the movement of the self-mobile device (such as removing higher point cloud data based on the height of the self-mobile device), and projecting it into a two-dimensional plane. Then extract the nearest point in the two-dimensional plane to convert the three-dimensional point cloud data into two-dimensional point cloud data. After completing the data conversion, the corresponding map update algorithm (such as the Bresenham algorithm) can be called based on the posture data of the corresponding data frame to update the grid map to determine the first navigation map perceived at the current moment.

[0075] In some embodiments, the aforementioned grid map may also be configured as a grid map in a three-dimensional space. In this case, the relevant algorithm may be an update algorithm for a three-dimensional grid map (such as a ray projection algorithm), which will not be described in detail here.

[0076] In some embodiments, considering that the subsequent loop optimization process is performed based on keyframes to optimize the pose data of the keyframes, in order to match the optimized pose data, the map can also be updated based on the pose data of the keyframes during the map update process.

[0077] In some embodiments, when updating a navigation map through a map update process, ordinary frames may be bound to key frames. Considering the timing of data frame acquisition, the key frame bound to each ordinary frame may be the key frame that precedes the ordinary frame and is closest to the ordinary frame in time. Thus, when performing the first navigation map update:

[0078] For ordinary frames: in response to the current frame being an ordinary frame, determine the binding key frame corresponding to the current frame and determine the relative posture of the current frame and the binding key frame, and update the first navigation map based on the relative posture of the current frame and the binding key frame and the point cloud data of the current frame.

[0079] For a key frame: in response to the current frame being a key frame, the first navigation map is updated based on the point cloud data and pose data of the current frame.

[0080] It should be noted that while the above description only involves the relative pose of regular frames and keyframes, actual updates can also be calculated based on this relative pose to determine the actual pose of the regular frame. The aforementioned representation primarily facilitates replacement after a keyframe's pose data changes. That is, after a keyframe's pose data changes, the actual pose of the regular frame can be adjusted based on this pose change and the relative pose of the two, eliminating the need for separate conversions, thereby improving processing speed.

[0081] The aforementioned S230 to S250 is a posture optimization process based on loop frames, wherein the loop frames can specifically refer to data frames of areas that the mobile device has passed through / perceived during its historical movement. That is, when a certain position is passed by the mobile device and then passes through the position again, the data frame corresponding to the moment of passing through the position again can be regarded as a loop frame, and the data frame corresponding to the moment of passing through the position in the past can be recorded as a matching frame.

[0082] That is, when a loop frame appears, it means that the environment in which the loop frame is located is perceived twice. Then, the actual pose error between the loop frame and the matching frame can be determined based on the point cloud data of the environment, thereby eliminating the cumulative error.

[0083] Considering that a self-moving device, when its posture remains unchanged, generally moves continuously in a specific direction and does not repeatedly pass through a certain location, loop frames often occur when the device's posture changes (such as turning around or circling). Therefore, to reduce the computational complexity of loop detection, only data frames with significant posture changes can be detected. The aforementioned normal frames can refer to data frames with minimal posture changes, while key frames can refer to data frames with significant posture changes.

[0084] In particular, the loop detection process in the present application can be triggered in response to a new key frame, that is, the conditional variable of the loop detection process can monitor the data update of the key frame cache partition of the loop detection process, so that when the key frame cache partition data is updated, the conditional variable is in a valid state to trigger the loop detection process to execute the aforementioned S230.

[0085] In S230 , when the current frame is a key frame, whether the current frame is a loop frame can be determined through a loop detection process, so that when the current frame is a loop frame, a matching frame of the current frame and a relative posture between the current frame and the matching frame can be determined.

[0086] In the aforementioned S230, the determination of the loop frame is generally based on the determination of the candidate frame based on the posture data, and then the matching frame is determined based on the point cloud data of the candidate frame. For example, the posture data can be accumulated in reverse with the current frame as the starting point to determine multiple candidate frames whose position integrals of the posture data are close to the current frame, and then the point cloud data of the candidate frame is aligned with the point cloud data of the current frame to determine the matching frame. Among them, when the aforementioned process cannot find a matching frame, it can be determined that the current frame is not a loop frame, thereby terminating the loop optimization process of the current frame and performing detection on the next data frame. For more information on determining loop frames, please refer to Figure 4 and its related descriptions.

[0087] In some embodiments, if the current frame is a loop frame, the loop result of the current frame can be stored in the loop result cache partition of the data cache space of the subsequent process. The loop result is generally stored in the loop result cache partition of the subsequent loop optimization process.

[0088] As mentioned above, the condition variable of the loop optimization process can monitor the data update status of the loop result cache partition of the loop optimization process, so that when the loop result cache partition data is updated (i.e., a loop data frame appears), the condition variable is put into a valid state to trigger the loop optimization process to execute the aforementioned S240.

[0089] In the aforementioned S240, the loop closure optimization process can call the loop closure result and correct the posture data of multiple target frames between the current frame and the matching frame based on the loop closure result to determine the corrected posture data of each target frame. Among them, the correction process can be performed based on a common posture correction algorithm (such as graph optimization, factor graph method, etc.). Specifically, the relative posture of the current frame and the matching frame is often configured as a constraint condition, so as to execute the corresponding posture correction algorithm to determine the corrected posture data of each key frame.

[0090] In some embodiments, to reduce the amount of computation, the loop closure optimization process can be constructed based on an incremental principle, that is, when a loop frame is detected, only the keyframes between the loop frame and its matching frame can be corrected, rather than a global correction. For more information about incremental nonlinear optimization, please refer to Figure 4 and its related descriptions.

[0091] In some embodiments, considering that the aforementioned correction process can update the pose data of the keyframe, after determining the corrected pose data, the corrected pose data can be synchronized to various data cache areas. To facilitate triggering subsequent processing, the correction process can be configured as an optimization record (which may include the modified keyframe ID) stored in the optimization record cache partition of the data cache area. The optimization record is generally stored in the optimization record cache partition of the map update process.

[0092] As mentioned above, the second condition variable of the map update process can monitor the data update status of the optimization record cache partition of the map update program, so that when the optimization record cache partition data is updated (i.e., a new optimization record appears), the second condition variable is put into a valid state, thereby triggering the map update process in the second thread to execute the aforementioned S250.

[0093] The second thread may refer to a different thread or thread combination within the processor than the aforementioned first thread. This means that map update processes in different threads can be executed in parallel, thereby enabling navigation map optimization based on loop frames without affecting the real-time update of the navigation map. The map update process of the first thread generally runs in the foreground, directly presenting the first navigation map. The map update process of the second thread generally runs in the background, and upon completion of the optimization, the second navigation map it determines replaces the aforementioned first navigation map.

[0094] The map update process of the second thread can be similar to the map update process of the first thread, thereby performing map modeling based on the corrected pose data to determine the second navigation map. Considering that the corrected pose data can be directly synchronized to each data cache area in the aforementioned steps, the map update process of the second thread can directly call each keyframe to construct the navigation map to determine the second navigation map, and use the second navigation map to replace the first navigation map after the reconstruction is completed.

[0095] In some embodiments, considering that the aforementioned loop optimization process can be configured as an incremental process, the map update process of the second thread can also be matched with the process and configured as an incremental map update process. Combined with the aforementioned keyframe-based navigation map update process, the incremental navigation map update process can be: first obtain the historical navigation map at the acquisition moment of the matching frame from the data cache area of ​​the map update process. Then determine the multiple bound ordinary frames bound to the target frame and the relative posture of the bound ordinary frame relative to the target frame. Finally, based on the corrected posture data of the multiple target frames and the relative posture of the bound ordinary frame and the target frame, the historical navigation map is updated to determine the second navigation map.

[0096] In addition, considering the first navigation point Figure 1 The data cache area of ​​the map update process may not be able to cache the data. In this case, the aforementioned process may also directly construct a navigation map based on the corrected key frames and ordinary frames to determine the second navigation map.

[0097] In some embodiments, considering the running time of the second thread, multiple data frames may be obtained from the mobile device during the actual determination of the second navigation map. The map update program of the second thread can update the second navigation map according to the newly obtained data frames until the second navigation map is synchronized with the first navigation map. For details, please refer to Figure 5 and its related descriptions.

[0098] Therefore, based on the above, a multi-threaded map update method and its storage medium, electronic device and program product are provided. The present application achieves the decoupling of the data relationship of each link in the map update method by caching data outside the process, and adjusts the conditional variables of the process by updating the cached data, thereby achieving the decoupling of the trigger logic between each link. Therefore, the present application can split the map update method into multiple independently running processes. In actual execution, each process can be executed in a different thread, thereby achieving multi-linear parallel update of the map to improve the execution speed of the map update. In particular, in order to eliminate the accumulated error in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be implemented by calling the same link in parallel on different threads, so that the elimination of the accumulated error in the map does not affect the normal display of the map.

[0099] Exemplary Data Caching Method

[0100] To further illustrate the aforementioned frame data caching process, the present application also provides an exemplary flow chart of a data caching method ( Figure 3 ).

[0101] like Figure 3 As shown, process P300 may include the following steps:

[0102] S310: Determine a posture change of the mobile device at the current moment based on the posture data of the current frame.

[0103] S320: Determine the frame type of the current frame based on the posture change.

[0104] S330 : Copy the current frame to a corresponding partition in the data cache area of ​​the data acquisition process based on the frame type of the current frame.

[0105] S340 : In response to the data update in the data cache area of ​​the data acquisition process, copy the current frame to the data cache area of ​​other processes.

[0106] In S310, the posture change of the self-mobile device at the current moment can generally be determined by the time series integration of the inertial sensor data. For example, the posture data of the frame data can be represented by the time series inertial sensor data of the frame data between the previous moment and the current moment. The posture change of the self-mobile device between the two moments can then be determined using an odometer calculation method.

[0107] As mentioned above, the frame type can generally be configured based on a threshold. When the posture change is less than the threshold, the current frame's frame type is configured as a normal frame; when the posture change is greater than the threshold, the current frame's frame type is configured as a key frame.

[0108] Considering that subsequent processing may involve processing both common frames and key frames simultaneously and needs to rely on specific data of key frames, after determining the frame type, the frame type identifier can be added to the data structure of the data frame.

[0109] In some embodiments, considering the aforementioned binding of normal frames to keyframes, the representation of the pose data in each frame data can be adjusted. The pose data of a keyframe can reflect its actual position and pose, while the pose data of a normal frame can reflect its relative position (also known as position transformation) and relative pose (also known as pose transformation) relative to the bound keyframe.

[0110] Frame data can be cached using dual buffering. Specifically, the data acquisition process can be configured with a data cache area, while subsequent processes can also be configured with data cache areas. Each data frame is stored in each data cache area. Consequently, when subsequent processes access data, they only access their own data cache area, not the data acquisition process's data cache area. This reduces inter-thread data calls and significantly reduces thread blocking.

[0111] To further avoid read-write conflicts, particularly those in the data cache area of ​​the data acquisition process, copying data from the data cache area of ​​the data acquisition process to the data cache areas of each process can be performed based on the data update status and read-write status of the data cache area of ​​the data acquisition process. Since this step is performed by the data acquisition process, it does not require conditional variables and can be executed directly by reading the corresponding status.

[0112] Furthermore, considering that there may be read-write conflicts during data replication, the data cache area may be configured with an occupied flag, and the aforementioned S340 may further include the following steps:

[0113] S341 . Adjust the occupation flag of the data cache area of ​​the data acquisition process based on the writing status of the data cache area of ​​the data acquisition process.

[0114] S342: In response to the change in the occupancy flag and the occupancy flag being in an idle state, determine that the data in the data cache area is updated and copy the current frame to the data cache area of ​​another process.

[0115] The occupancy flag reflects the current read and write status of the data cache area. The occupancy flag includes both write and idle states. To prevent simultaneous read and write requests to the data cache area, the data cache area cannot be read when the occupancy flag is in the write state.

[0116] Furthermore, it should be noted that the write status in the aforementioned S341 can include data written to disk during the data acquisition process, or can also include modifications to the posture data in subsequent steps. Therefore, based on the aforementioned S341, when the occupancy flag is adjusted, it can be determined that new data (such as the current frame or revised posture data) has been written to the data, thereby determining that the corresponding partition data has been updated, thereby triggering the subsequent steps.

[0117] It should be noted that, considering that read-write conflicts may also occur in the data cache area of ​​other processes, an occupation flag can also be configured for it based on the above process.

[0118] Exemplary loop correction process

[0119] To further illustrate the loop correction process provided by this application, this application also provides a signaling diagram of the loop correction process ( Figure 4 ). Figure 4 The signaling diagram shown can reflect the data and signal transmission relationship between the three parallel processes. Figure 4 The cross-thread trigger in can be understood as a process trigger based on condition variables.

[0120] Combining the above content with Figure 4 As can be seen from the process P400 shown, when the loop detection process is triggered by a key frame, S411 to S415 in P400 can be executed to determine the loop result and trigger the loop optimization process, thereby executing S421 to S424 in P400, thereby determining the loop optimization result and triggering the map update process of the second thread, thereby executing S431 to S433 in P400.

[0121] That is Figure 4 As shown in P400, the loop detection process can execute the following steps in response to the newly added key frame:

[0122] S411 : Determine a posture search tree for each key frame based on the posture data of multiple key frames.

[0123] S412: Match the posture data of the current frame with the posture search tree of each key frame, and determine at least one candidate frame from the multiple key frames.

[0124] S413: Determine a plurality of three-dimensional networks of different resolutions for at least one candidate frame.

[0125] S414 , performing point cloud registration on each 3D network based on the point cloud data of the current frame, and determining the registration result of the current frame.

[0126] S415. In response to the registration result including the target candidate frame in which each three-dimensional network matches the current frame, determine that the current frame is a loop frame and the target candidate frame is a matching frame, and determine the relative pose of the current frame and the matching frame.

[0127] Based on the above process, the loop closure optimization process can perform the following steps in response to the loop closure results:

[0128] S421. Call multiple target frames from the key frame cache partition.

[0129] S422: Determine multiple relative factor graphs based on multiple target frames.

[0130] S423: Determine a loop closure factor graph based on the relative pose between the current frame and the matching frame.

[0131] S424: Perform nonlinear optimization on the pose data of the multiple target frames based on the multiple relative factor graphs and the loop closure factor graphs to determine the loop closure optimization results.

[0132] Based on the above process, the map update process of the second thread can perform the following steps in response to the loop optimization result:

[0133] S431. Call multiple target normal frames from the normal frame buffer partition of the map update process based on the loop optimization result.

[0134] S432 : Determine the corrected pose data of the target ordinary frame based on the binding relationship between the target ordinary frame and the target frame and the corrected pose data of the target frame.

[0135] S433: Determine a second navigation map based on the corrected pose data of the multiple target frames and the corrected pose data of the multiple target common frames.

[0136] The aforementioned S411 to S412 can be understood as a process of determining candidate frames for the current frame from the key frames. Specifically, in the aforementioned S411 to S412, the determination of candidate frames can be achieved through a posture search tree. The posture search tree can be a tree-like search structure constructed based on posture data.

[0137] For example, the posture search tree can be a KDTree (K-dimensional binary search tree) constructed based on posture data (e.g., three-degree-of-freedom position). The current frame and multiple historical key frames can be first called from the key frame cache partition, thereby constructing a KDTree based on the posture data of the historical key frames. All key frames that are close to the current frame are then found in the KDTree. The construction and search of the KDTree can be found in the description of the relevant technology and will not be elaborated here.

[0138] In some embodiments, to prevent the current frame from being defined as a loop frame because it finds the area it just passed through, the identification information of the candidate frame determined above may satisfy a preset interval relationship with the identification information of the current frame. For example, the ID (or timestamp) of the candidate frame may be separated from the corresponding identification information of the current frame by at least a preset number of data frames (or a preset time interval).

[0139] After determining multiple candidate frames, S413 to S415 may be executed to determine whether the current frame is a loop frame. If the current frame is not a loop frame, the current process is terminated directly. If it is a loop frame, the loop result is determined.

[0140] Specifically, after obtaining a candidate frame, the point cloud information of several key frames near the candidate frame is combined to construct a local map. This local map is then aligned with the 3D point cloud of the current frame. Considering the high computational complexity of point cloud configuration, the local map can be sampled at different resolutions to determine 3D networks of different resolutions. These 3D networks at each resolution are then aligned with the point cloud data of the current frame.

[0141] The aforementioned configuration process can be performed incrementally at low resolution, with any unregistered local maps eliminated during the registration process to determine a target candidate frame whose 3D networks match the current frame. At this point, the current frame can be determined to be a loop frame, and the target candidate frame to be a matching frame.

[0142] Considering that the registration process of point cloud data and the aforementioned three-dimensional network often requires spatial movement, the relative position of the current frame and the matching frame can be determined by the registration relationship between the current frame point cloud data and the high-resolution three-dimensional network of the target candidate frame and the position of the mobile device in the three-dimensional network.

[0143] From the above steps S421 to S424 , it can be seen that the above loop process can be characterized as an incremental factor graph method.

[0144] In steps S421-S424 above, the factor graph method is an important optimization technique. Specifically, the factor graph method represents and processes data by constructing an undirected graph model consisting of variable nodes and factor nodes. Variable nodes represent system state parameters, such as pose transformations; factor nodes represent observation constraints or measurement functions. Based on the factor graph, state parameters (such as pose data) can be optimized.

[0145] The aforementioned incremental factor graph method can be understood as a method that introduces a loop closure factor graph into conventional constraints. Specifically, the incremental factor graph method can be performed on the key frame (denoted as the target frame) between the loop closure frame and the matching frame. By introducing the constraints of the loop closure frame into the general factor graph constraints of the target frame, nonlinear optimization can be performed.

[0146] Based on the aforementioned incremental factor graph method, multiple relative factor graphs can be first determined based on multiple target frames. A loop closure factor graph can then be determined based on the relative pose between the current frame and the matching frame. Finally, the pose data of the multiple target frames are nonlinearly optimized based on the multiple relative factor graphs and the loop closure factor graph to determine the corrected pose data for the multiple target frames. The relative factor graph reflects the relative pose between two consecutive target frames from the mobile device, and the loop closure factor graph can reflect the relative pose between the current frame (loop closure frame) and the matching frame.

[0147] Specifically, during the loop closure optimization process, the aforementioned relative factor graph can be determined as the keyframe cache partition is updated. That is, the loop closure optimization process can determine the relative pose between the newly added keyframe and the previous keyframe as the keyframe cache partition is updated, thereby constructing the relative factor graph.

[0148] The aforementioned loop closure factor graph can be determined based on the update of the cache area of ​​the aforementioned loop closure results. That is, the loop closure results (i.e., the loop closure frame, the matching frame of the loop closure frame, and the relative pose between the two) can be obtained from the cache area. Therefore, a new real constraint can be generated based on the relative position and added to the factor graph (referred to as the loop closure factor graph).

[0149] Different from the above process, in traditional factor graphs, a strong prior factor is often added to the first frame, so that global optimization can be performed during loop closure optimization to eliminate the prior factor. In this application, the relationship between key frames can be reflected through the relative factor graph, and the actual corresponding relationship can be reflected through the loop closure factor graph. This eliminates the need for prior factors and the need to reconstruct the factor graph, thus realizing an incremental factor graph method that eliminates cumulative errors while reducing computational overhead.

[0150] Based on the aforementioned updated factor graph, incremental nonlinear optimization can be performed based on the factor graph to determine the optimized corrected pose data of the target frame between the loop frame (current frame) and the matching frame.

[0151] Conventional nonlinear optimization methods require global optimization of all variables in the entire system each time new data is added. However, the incremental nonlinear optimization method provided by this application optimizes only the local variables affected by the new data each time, rather than recalculating the entire system. This significantly improves computational efficiency. Specifically, for larger maps, incremental nonlinear optimization can complete optimization in just tens of milliseconds.

[0152] As can be seen from the aforementioned S431 to S433, when actually updating the map, the map update process of the second thread is similar to that of the first thread. The specific difference is that the map update process of the second thread often needs to update multiple data frames at a time. In actual application, each data frame can be updated one by one.

[0153] Considering the large amount of data in the second thread's map update process, which often requires a certain amount of time to process, and the fact that a loop frame often occurs, multiple loop frames often appear in succession, the second thread's map update process can be adjusted based on actual conditions.

[0154] In some embodiments, if other loop frames appear before the map update process of the thread completes the update, the current map update may be terminated, so that the map is updated based on the latest loop frame.

[0155] In some embodiments, to avoid the possibility of repeated termination of the second thread's map update process, which could ultimately lead to a failure to correct the problem, a third thread can trigger another map update process when a new loop frame appears. The second thread's map update process remains uninterrupted, and the third thread's map update process can be retriggered in response to the new loop frame. This ensures that navigation map corrections can at least be implemented based on the second thread's map update process.

[0156] Exemplary first navigation map correction method

[0157] In some embodiments, to further illustrate the process of replacing the first navigation map with the second navigation map, the present application also provides an exemplary flow chart of a method for correcting the first navigation map ( Figure 5 ).

[0158] like Figure 5 As shown, process P500 may include the following steps:

[0159] S510: Initialize and call a map update process in the second thread to determine a background map update process.

[0160] S520: Determine a second navigation map based on the corrected pose data of the plurality of target frames through a background map update process.

[0161] S530: Determine whether the second navigation map is synchronized with the first navigation map.

[0162] S540: Replace the first navigation map with the second navigation map, and terminate the background map update process.

[0163] S550: Retrieve a difference data frame between the first navigation map and the second navigation map from the data cache area, and update the second navigation map based on the difference data frame.

[0164] S540 is the "yes branch" of S530, and S550 is the "no branch" of S530. In particular, after executing S550, S530 may be executed again to form a loop processing process until the second navigation map is synchronized with the first navigation map.

[0165] In some embodiments, whether the second navigation map is synchronized with the first navigation map can be determined based on whether the latest data frames in each navigation map are identical. If they are not identical, it indicates that the first navigation map received a new data frame during the update of the second navigation map. To ensure continuous use of the map, it is necessary to execute S550 to update the missing difference data in the second navigation map.

[0166] Exemplary devices

[0167] Combined with the above Figures 1 to 5 , described in detail the method embodiment of the present application, and the device embodiment of the present application is described in detail below. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0168] The present application also provides a multi-threaded map update system, including functional modules for implementing the multi-threaded map update method provided by the present application. A three-dimensional map update system can be integrated into the operating environment of an electronic device, thereby providing a three-dimensional map update function for the electronic device.

[0169] For example Figure 6 As shown, Figure 6 3D map updating system 600 may include a data acquisition module 610, a first map updating module 620, a loop detection module 630, a loop optimization module 640, and a second map updating module 650.

[0170] The data acquisition module 610 can be used to obtain the current frame at the current moment through the data acquisition process, and copy the current frame to the corresponding data cache area based on the frame type of the current frame, wherein the current frame includes the point cloud data and posture data at the current moment, the frame type includes key frames and ordinary frames, and the data cache area includes ordinary frame cache partitions and key frame cache partitions.

[0171] The first map update module 620 can be used to wake up the map update process in the first thread in response to the first condition variable of the map update process satisfying the process wake-up condition, call the current frame from the data cache area of ​​the map update process through the map update process, and update the first navigation map of the mobile device based on the current frame, wherein the first condition variable of the map update process reflects the data update status of the key frame cache partition and the ordinary frame cache partition of the map update program.

[0172] The loop detection module 630 can be used to wake up the loop detection process in response to the condition variable of the loop detection process satisfying the process wake-up condition, call the current frame and multiple key frames from the key frame cache partition of the loop detection process through the loop detection process, determine the loop result of the current frame based on the multiple key frames and the current frame, and store it in the loop result cache partition of the data cache area, wherein, when the current frame is a loop frame, the loop result includes the current frame, the matching frame corresponding to the current frame, and the relative posture between the current frame and the matching frame, and the condition variable of the loop detection process reflects the data update status of the key frame cache partition of the loop detection process.

[0173] The loop closure optimization module 640 can be used to wake up the loop closure optimization process in response to the conditional variables of the loop closure optimization process satisfying the process wake-up conditions, and determine the loop closure optimization results of multiple target frames between the current frame and the matching frame based on the loop closure results through the loop closure optimization process and store them in the optimization record cache partition of the data cache area, wherein the conditional variables of the loop closure optimization process reflect the data update status of the loop closure result cache partition of the loop closure optimization process, and the loop closure optimization results include corrected pose data.

[0174] The second map update module 650 can be used to wake up the map update process on the second thread in response to the second condition variable of the map update process satisfying the process wake-up condition, and determine the second navigation map of the mobile device based on the loop optimization results of multiple target frames through the map update process and replace the first navigation map, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

[0175] The present application also provides an electronic device, wherein the electronic device can be specifically configured as the aforementioned Figure 1 wherein, in actual application, the electronic device can be used as a user terminal to implement the multi-threaded map update method provided in the present application. For example, the sensor on the electronic device can determine the data frame (including posture data and point cloud data) at each moment and update the first navigation map. wherein, the electronic device can correct the posture data when the frame data at the current moment is a loop frame, thereby determining the second navigation map based on the corrected posture data to replace the first navigation map.

[0176] The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction processor in the computer program product provided by the present application is executed, the multi-threaded map updating method provided by the present application can be implemented.

[0177] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0178] It should be understood that the term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0179] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an operation and maintenance management (OAM), or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disk (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)). The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0184] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A multi-threaded map updating method, characterized in that: The multi-threaded map updating method is applied to a mobile device, including: Acquire a current frame at a current moment through a data acquisition process, and copy the current frame to a corresponding data cache area based on a frame type of the current frame, wherein the current frame includes point cloud data and pose data at the current moment, the frame type includes a key frame and a normal frame, and the data cache area includes a normal frame cache partition and a key frame cache partition; In response to a first condition variable of a map update process satisfying a process wake-up condition, waking up the map update process in a first thread, calling the current frame from a data cache area of ​​the map update process through the map update process, and updating the first navigation map of the mobile device based on the current frame, wherein the first condition variable of the map update process reflects data update status of a key frame cache partition and a common frame cache partition of the map update program; In response to a condition variable of a loop detection process satisfying a process wake-up condition, waking up the loop detection process, calling the current frame and multiple key frames from a key frame cache partition of the loop detection process through the loop detection process, determining a loop result of the current frame based on the multiple key frames and the current frame and storing the result in a loop result cache partition of the data cache area, wherein, when the current frame is a loop frame, the loop result includes the current frame, a matching frame corresponding to the current frame, and a relative pose between the current frame and the matching frame, and the condition variable of the loop detection process reflects a data update status of the key frame cache partition of the loop detection process; In response to a condition variable of the loop closure optimization process satisfying a process wake-up condition, waking up the loop closure optimization process, determining, by the loop closure optimization process, loop closure optimization results of a plurality of target frames between the current frame and the matching frame based on the loop closure result and storing the results in an optimization record cache partition of the data cache area, wherein the condition variable of the loop closure optimization process reflects a data update status of the loop closure result cache partition of the loop closure optimization process, and the loop closure optimization result includes corrected pose data; In response to a second condition variable of a map update process satisfying a process wake-up condition, the map update process is woken up in a second thread, and a second navigation map of the mobile device is determined by the map update process based on loop optimization results of the multiple target frames and replaces the first navigation map, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

2. The multi-threaded map updating method according to claim 1, characterized in that: The copying of the current frame to a corresponding data buffer area based on the frame type of the current frame includes: Determining a posture change of the mobile device at a current moment based on the posture data of the current frame; determining a frame type of the current frame based on the posture change, wherein when the posture change is less than a threshold, the frame type of the current frame is configured as a normal frame; when the posture change is greater than the threshold, the frame type of the current frame is configured as a key frame; Copying the current frame to a corresponding partition in the data cache area of ​​the data acquisition process based on the frame type of the current frame; In response to data update in the data cache area of ​​the data acquisition process, the current frame is copied to the data cache area of ​​other processes.

3. The multi-threaded map updating method according to claim 2, characterized in that: In response to the data update of the data cache area, copying the current frame to the data cache area of ​​another process includes: Adjusting an occupancy flag of a data cache area of ​​the data acquisition process based on a write condition of the data cache area of ​​the data acquisition process, wherein the occupancy flag includes a write state and an idle state; In response to the change in the occupancy flag and the occupancy flag being the idle state, it is determined that the data in the data cache area is updated and the current frame is copied to the data cache area of ​​another process.

4. The multi-threaded map updating method according to claim 1, characterized in that: Determining loop closure optimization results of a plurality of target frames between the current frame and the matching frame based on the loop closure result through the loop closure optimization process includes: Recalling the plurality of target frames from the key frame buffer partition; Determining a plurality of relative factor graphs based on the plurality of target frames, wherein the relative factor graphs reflect relative poses between two consecutive target frames of the self-mobile device; Determine a loop closure factor graph based on the relative pose between the current frame and the matching frame; Nonlinear optimization is performed on the pose data of the multiple target frames based on the multiple relative factor graphs and the loop closure factor graphs to determine the loop closure optimization result.

5. The multi-threaded map updating method according to claim 4, characterized in that: Determining the second navigation map of the mobile device based on the loop optimization results of the plurality of target frames through the map update process includes: Based on the loop optimization result, calling a plurality of target normal frames from the normal frame buffer partition of the map update process, wherein the plurality of target normal frames include normal frames between the current frame and the matching frame; Determining the corrected pose data of the target ordinary frame based on the binding relationship between the target ordinary frame and the target frame and the corrected pose data of the target frame; A second navigation map is determined based on the corrected pose data of the plurality of target frames and the corrected pose data of the plurality of target ordinary frames.

6. The multi-threaded map updating method according to claim 1, characterized in that: Waking up the map update process in the second thread, and determining the second navigation map of the mobile device based on the loop optimization results of the multiple target frames through the map update process and replacing the first navigation map, includes: Initializing a map update program of the map update process in the second thread, and determining the map update process of the second thread; determining the second navigation map based on the corrected pose data of the plurality of target frames; determining whether the second navigation map is synchronized with the first navigation map; If so, replacing the first navigation map with the second navigation map and terminating the map update process of the second thread; If not, a difference data frame is retrieved from the data cache area of ​​the map update process, and the second navigation map is updated based on the difference data frame, wherein the difference data frame reflects each data frame of the second navigation map that is not updated after loop optimization.

7. The multi-threaded map updating method according to claim 1, characterized in that: The determining a loop result of the current frame based on the multiple key frames and the current frame includes: Determine a pose search tree for each key frame based on the pose data of the plurality of key frames; Matching the pose data of the current frame with the pose search trees of the key frames, and determining at least one candidate frame from the multiple key frames; determining a plurality of three-dimensional networks of different resolutions for the at least one candidate frame; Performing point cloud registration on each three-dimensional network based on the point cloud data of the current frame to determine a registration result of the current frame; In response to the registration result including a target candidate frame whose respective three-dimensional networks match the current frame, the current frame is determined to be a loop frame, the target candidate frame is the matching frame, and a relative pose between the current frame and the matching frame is determined.

8. The multi-threaded map updating method according to claim 1, characterized in that: The calling of the current frame from the data cache area of ​​the map update process by the map update process and updating the first navigation map of the mobile device based on the current frame includes: In response to the current frame being a common frame, determining a bound keyframe corresponding to the current frame and determining a relative pose between the current frame and the bound keyframe, and updating the first navigation map based on the relative pose between the current frame and the bound keyframe and point cloud data of the current frame; In response to the current frame being a key frame, the first navigation map is updated based on the point cloud data and pose data of the current frame.

9. The multi-threaded map updating method according to claim 8, characterized in that: Determining a second navigation map from the mobile device based on loop optimization results of a plurality of target frames through the map update process includes: Determining a historical navigation map at the time of acquisition of a matching frame from a data cache area of ​​the map update process; Determining a plurality of bound common frames bound to the target frame and relative positions of the bound common frames relative to the target frame; The historical navigation map is updated based on the corrected pose data of multiple target frames and the relative pose between the bound common frame and the target frame to determine the second navigation map.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction processor is executed, the multi-threaded map updating method according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the multi-threaded map updating method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the multi-threaded map updating method according to any one of claims 1 to 9.

Images