Multi-thread map updating method, storage medium, electronic equipment and program product

Abstract

The invention provides a multi-thread map updating method, a storage medium, electronic equipment and a program product. Through data caching outside the process, decoupling of data relations of all links in the map updating method is achieved, condition variables of the process are adjusted through updating of cached data, and decoupling of trigger logic among all links is achieved. Therefore, the map updating method can be divided into a plurality of processes which run independently, and each process can be executed in different threads in actual execution, so that multilinear parallel updating of the map is realized, and the execution speed of map updating is improved. Particularly, in order to eliminate accumulated errors in the map, whether the current frame collected at the current moment is a loopback frame or not can be judged. When the current frame is the loopback frame, map correction based on the loopback frame and map updating based on the current frame can be realized at the same time by calling the same functional module in parallel in different threads, so that elimination of accumulated errors in the map does not affect normal display and updating of the map.

Description

Technical Field

[0001] The present application relates to the technical field of mobile equipment, and in particular to a multi-threaded map updating method, storage medium, electronic device and program product. Background Art

[0002] Autonomous devices can be machines and devices that can move by themselves, such as embodied robots, cleaning robots, electric vehicles, etc. During the movement of the autonomous device, the autonomous device can generally perceive the external environment and determine its own movement based on the sensors integrated on the autonomous device, thereby building a perception map of the autonomous device and performing subsequent work.

[0003] Considering the computing power of the mobile device itself and the complexity of the sensor data, how to improve the processing speed of map data is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a multi-threaded map update method, storage medium, electronic device and program product, which can decouple the various functions within the map update algorithm through an out-of-process data cache, so that they can be executed simultaneously in different threads, thereby realizing a multi-threaded map update to speed up the map update process.

[0005] In a first aspect, the present application provides a multi-threaded map updating method, which is applied to a mobile device, including:

[0006] The current frame at the current moment is obtained through the data acquisition process, and the current frame is copied to the corresponding data cache area based on the frame type of the current frame, wherein the current frame includes the point cloud data and posture data at the current moment, the frame type includes key frames and ordinary frames, and the data cache area includes ordinary frame cache partitions and key frame cache partitions.

[0007] In response to the first condition variable of the map update process satisfying the process wake-up condition, the map update process is woken up in the first thread, the current frame is called from the data cache area of ​​the map update process through the map update process, and the first navigation map of the mobile device is updated based on the current frame, wherein the first condition variable of the map update process reflects the data update status of the key frame cache partition and the ordinary frame cache partition of the map update program.

[0008] In response to the conditional variable of the loop detection process satisfying the process wake-up condition, the loop detection process is woken up, and the current frame and multiple key frames are called from the key frame cache partition of the loop detection process through the loop detection process. The loop result of the current frame is determined based on the multiple key frames and the current frame and stored in the loop result cache partition of the data cache area, wherein, when the current frame is a loop frame, the loop result includes the current frame, the matching frame corresponding to the current frame, and the relative posture between the current frame and the matching frame, and the conditional variable of the loop detection process reflects the data update status of the key frame cache partition of the loop detection process.

[0009] In response to the conditional variables of the loop closure optimization process satisfying the process wake-up conditions, the loop closure optimization process is woken up, and the loop closure optimization results of multiple target frames between the current frame and the matching frame are determined by the loop closure optimization process based on the loop closure results and stored in the optimization record cache partition of the data cache area, wherein the conditional variables of the loop closure optimization process reflect the data update status of the loop closure result cache partition of the loop closure optimization process, and the loop closure optimization results include corrected pose data.

[0010] In response to the second condition variable of the map update process satisfying the process wake-up condition, the map update process is woken up in the second thread, and the second navigation map of the mobile device is determined based on the loop optimization results of multiple target frames through the map update process and replaces the first navigation map, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

[0011] In a second aspect, the present application provides an electronic device. The electronic device includes: at least one processor and a memory connected to the at least one processor. The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-threaded map update method described in the first aspect.

[0012] In a third aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when the computer program / instruction processor is executed, implements the multi-threaded map updating method as described in the first aspect.

[0013] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program / instruction. When the computer program / instruction is executed by a processor, the multi-threaded map update method described in the first aspect is implemented.

[0014] The present application provides a multi-threaded map updating method, storage medium, electronic device and program product. It relates to the field of robotics technology. The present application achieves the decoupling of data relations among various functional modules in the map updating method by caching data outside the process, and adjusts the conditional variables of the process by updating the cached data, thereby achieving the decoupling of the trigger logic among various functional modules. Therefore, the present application can split the map updating method into multiple independently running processes, and in actual execution, each process can be executed in a different thread, thereby achieving multi-linear parallel updating of the map to improve the execution speed of the map update. In particular, in order to eliminate the accumulated errors in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be simultaneously implemented by calling the same functional module in parallel in different threads, so that the elimination of the accumulated errors in the map does not affect the normal display and update of the map. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 This is a schematic diagram of functional modules of a mobile device provided in some embodiments of the present application.

[0017] Figure 2 This is an exemplary flowchart of a map updating method provided in some embodiments of the present application.

[0018] Figure 3 This is an exemplary flowchart of a data caching process provided in some embodiments of the present application.

[0019] Figure 4 It is a schematic diagram of the loop correction process provided by some embodiments of the present application.

[0020] Figure 5 This is an exemplary flowchart of the first navigation map correction method provided in some embodiments of the present application.

[0021] Figure 6 This is a system module diagram of a multi-threaded map update system provided in some embodiments of the present application. DETAILED DESCRIPTION

[0022] Application Overview

[0023] A self-moving device may refer to an electronic device equipped with a mobile component. Among them, the self-moving device can achieve the corresponding work task through its own movement. Its specific work task and movement mode can be determined based on the specific form of the self-moving device. For example, for cleaning robots (such as sweeping robots, lawn mowing robots, etc.), their movement mode generally depends on the moving device on the chassis (such as tires, tracks, etc.), and their work tasks are generally performed based on their integrated cleaning devices (such as vacuum devices, grass cutting devices, etc.). For another example, for embodied robots (such as humanoid robots, bionic robots, etc.), their movement mode is generally also based on their integrated moving devices (such as humanoid mechanical feet, tires, etc.), and their work tasks are generally performed based on robot arms or other integrated devices.

[0024] In actual operation, self-mobile devices often integrate SLAM (Simultaneous Localization And Mapping) technology and are considered a visual SLAM system. SLAM is a technology used for robots, self-driving vehicles and other equipment, which aims to achieve real-time positioning and map construction of equipment in unknown environments. Its core task is to obtain environmental information through sensor data, and use this information to determine the position and direction of the device in the environment, while gradually building a map of the environment.

[0025] SLAM technology mainly relies on a variety of sensors to collect environmental information, including cameras, Lidar, inertial measurement units (IMU), etc. The data provided by these sensors can be used to estimate the device's pose or transformation after being processed by algorithms, that is, the device's position and posture in the world coordinate system.

[0026] However, in the visual SLAM system of self-mobile equipment, the estimation of pose is often a recursive process, that is, the pose of the current frame is solved by the pose of the previous frame, so the error is passed on from frame to frame, resulting in cumulative error.

[0027] In the related art, loop closure detection can be used to overcome the aforementioned cumulative error. The basic principle of loop closure detection is to detect whether the current frame forms a loop with a previous key frame. Specifically, when a key frame is inserted, the loop closure detection thread is started to check whether the currently inserted key frame forms a loop with the previous key frame. If so, the map points and postures corresponding to these frames are retrieved in order according to the queried key frame ID and the current frame ID, and the optimization function is executed.

[0028] For example, in a visual SLAM system, continuous image frames are obtained through a camera, the camera's position and posture in the environment map are calculated in real time, key frames are selected at regular intervals and saved in a key frame queue, and the key frame posture is saved in the map information. In the loop detection task, for the current image frame Iq, in the image queue Id = {In, n = 1, 2, ..., N} with N key frames, the matching key frame Id is found, its relative rotation qR and translation t are calculated, and the actual posture of the current frame in the environment map is obtained after calculation with the saved posture of qId, thereby correcting the previously calculated accumulated error (such as adjusted posture data, thereby updating the map).

[0029] The present application has found that in the related art, the aforementioned process is often implemented in a self-mobile device based on the same program, that is, in a common map update algorithm, loop detection can be first performed based on the received sensor data, and loop optimization can be performed when the currently collected data frame is a loop frame, and the map data can be reconstructed based on the loop optimization result.

[0030] Further considering the computational complexity of loop detection, loop optimization, and map reconstruction, in actual applications, when executing the aforementioned processes, obvious lags may occur during loop optimization, seriously affecting the working conditions of the mobile device.

[0031] Based on the above situation, in order to improve the operating efficiency of relevant algorithms, the present application creatively decouples the internal data dependency and trigger relationship dependency of the aforementioned algorithms, and decomposes the entire optimization process into multiple processes running independently in different threads, thereby performing map updates in a multi-threaded form.

[0032] Specifically, the present application achieves the decoupling of data relationships between various links in the map update method by caching data outside the process, and achieves the decoupling of trigger logic between various links by adjusting the conditional variables of the process through the update of cached data. Therefore, the present application can split the map update method into multiple independently running processes, and in actual execution, each process can be executed in a different thread, thereby achieving multi-linear parallel update of the map to improve the execution speed of map update. In particular, in order to eliminate the accumulated errors in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be implemented by calling the same link in parallel on different threads, so that the elimination of accumulated errors in the map does not affect the normal display of the map.

[0033] To further describe the self-moving device that implements the aforementioned technology, the present application provides a functional module diagram of the self-moving device ( Figure 1 ).

[0034] In the present application, the self-moving device 100 may be an electronic device with a self-moving function. Specifically, the self-moving device may also be a semi-self-moving device or a fully autonomous mobile device. For example, the self-moving device 100 may be an embodied robot, a cleaning robot, an electric vehicle, etc. In some embodiments, the self-moving device 100 may have an operating capability, then it is generally integrated with a corresponding operating device (such as a mowing rope, a robotic arm, a storage bin, etc.), which may be adjusted according to actual conditions, and this application will not elaborate on this.

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the multi-threaded map update method and the related self-mobile device in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] Exemplary multi-threaded map update method

[0040] To illustrate the multi-threaded map updating method provided by the present application, the present application provides a module schematic diagram of a mobile device ( Figure 1 ) and an exemplary flowchart of a multi-threaded map updating method ( Figure 2 ). Figure 1 , Figure 2 The multi-threaded map updating method provided by this application is described in detail.

[0041] In accordance with the foregoing, this application Figure 1The self-mobile device 100 in the present invention is generally integrated with the aforementioned visual SLAM system. That is, the self-mobile device 100 can perform real-time positioning and map construction in an unknown environment, thereby performing corresponding operation tasks.

[0042] like Figure 1 As shown, the self-moving device 100 may include a processor 110, a memory 120 and a sensor 130. The processor 110 may be connected to the aforementioned memory 120 and the sensor 130 for communication. In addition, the self-moving device 100 is often also configured with a mobile device (such as tires, robot feet, etc.) for driving the self-moving device 100 to move and an operating device (such as a robot arm, a warehouse, etc.) for performing operations. The mobile device and the operating device can be configured according to actual needs, and will not be described in detail here.

[0043] The processor 110 may refer to a collection of devices having computing capabilities in the mobile device 100. For example, the processor 110 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or a processor, or any other conventional processor, etc.

[0044] The memory 120 may refer to a collection of devices having data storage capabilities in the mobile device 100. For example, the memory 120 may include a random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0045] The sensor 130 may be a collection of devices for acquiring sensor data at each moment. Among them, the sensor 130 may include at least an environmental sensor 131 for sensing the environment and a posture sensor 132 for sensing the posture of the self-mobile device 100. For example, the aforementioned environmental sensor 131 may generally include a radar sensor (such as a laser radar, a millimeter wave radar, etc.) and / or a camera sensor (such as a multi-eye camera, a depth of field camera, etc.). The environmental sensor 131 can sense the environment in which the self-mobile device 100 is located to determine its point cloud data. The aforementioned posture sensor 132 may generally be configured as an inertial sensor, so as to determine the inertial information at each moment to reflect the posture of the self-mobile device (such as performing time series integration).

[0046] In some embodiments, during the actual data collection process, the environmental sensor 131 and the posture sensor 132 can be synchronized in time and space, that is, point cloud data and posture data are collected at the same time and packaged into a data frame at the current time.

[0047] In the actual map update process, the memory 120 may store instructions related to the map update algorithm. When the instructions are executed by the processor 110, the processor 110 can execute the multi-threaded map update method provided in the present application.

[0048] Considering that the map updating method provided by the present application is multi-threaded, the aforementioned processor 110 has multi-threaded processing capability, and the aforementioned memory 120 stores different links of the map updating method in different programs. Therefore, when executing the multi-threaded map updating method provided by the present application, different programs can be called through different threads to generate corresponding processes, thereby realizing map updating in the form of multi-threaded parallel execution.

[0049] Specifically, the programs in the aforementioned memory 120 may mainly include a data acquisition program, a map update program, a loop detection program, and a loop optimization program. Considering the current architecture of the visual SLAM system, the aforementioned data acquisition program may be a front-end tracking program of the visual SLAM system, and the map update program, the loop detection program, and the loop optimization program may be a back-end optimization program of the visual SLAM system.

[0050] In actual applications, the data collection program can provide collected data frames for each backend optimization program. That is, in a multi-threaded map update, the data collection program is configured as a producer, and the three backend optimization programs can be configured as consumers.

[0051] When each program (i.e., instruction set) in the aforementioned memory 120 is called by the processor 110, it can be executed on different threads to form processes of multiple different threads. Among them, after the data acquisition program is called, a data acquisition process can be formed to determine the data frame at each moment (wherein the data frame at the current moment is recorded as the current frame). After the map update program is called, a map update process can be formed to update / determine the navigation map of the mobile device based on the data frame. After the loop detection program is called, a loop detection process can be formed to determine the loop result. After the loop optimization program is called, a loop optimization process can be formed to optimize the pose data of the relevant data frame based on the loop result. The specific processing process of each process can be found in the subsequent description, which will not be repeated here.

[0052] In some embodiments, considering that the aforementioned programs are generally integrated into a large program in traditional methods, in order to achieve relative independence of each link, the present application creatively decouples the data between the programs by constructing a data cache area, and configures conditional variables based on the data update status of the cache area, thereby achieving decoupling of the correlation between the various links, so that the traditional single program is converted into multiple independent programs, and can be executed in different threads, realizing multi-linear parallel map updates. That is, each back-end optimization program can be configured with a conditional variable, which can monitor the data update status of the corresponding cache area and is used to adjust the triggering / blocking status of the corresponding process based on its specific value.

[0053] In some embodiments, considering the principle of loop optimization, the data cache area of ​​each process may include a key frame cache partition and a normal frame cache partition. In addition, considering the correlation between each link, the data cache area of ​​each process may also include a cache partition of the output result of the previous link. For example, the data cache partition of the aforementioned loop optimization program may include a loop result cache partition for storing the loop results in the loop detection program, and the aforementioned map update program may include an optimization record cache partition for storing the loop optimization results of the loop optimization program.

[0054] Then the aforementioned conditional variables can be configured based on the cache partition of the corresponding process. Specifically, the map update program can be configured with a first conditional variable and a second conditional variable, wherein the first conditional variable can be used to reflect the data update status of the key frame cache partition and the ordinary frame cache partition of the map update program. The second conditional variable can be used to reflect the data update status of the optimized record cache partition. The conditional variables of the loop detection program can be used to reflect the data update status of the key frame cache partition. The conditional variables of the loop optimization program can be used to reflect the data update status of the loop result cache partition. The specific triggering logic of each conditional variable can be found in the subsequent content and will not be repeated here.

[0055] In some embodiments, the aforementioned data acquisition program may also be configured with a data cache area, thereby realizing a two-layer cache. That is, when executing the multilinear map update method of the present application, each process will only access its own data cache area, and will not access the data cache area of ​​the data acquisition program, thereby greatly reducing thread blocking.

[0056] Based on the above Figure 1 The relevant contents of have described the hardware and software basis for executing the map updating method of this application. Figure 2 The actual execution process is described. Figure 2 The process P200 shown can be executed by the aforementioned self-equipping device.

[0057] like Figure 2 As shown, P200 may include the following steps:

[0058] S210, acquiring a current frame at a current moment through a data acquisition process, and copying the current frame to a corresponding data buffer area based on a frame type of the current frame.

[0059] S220, in response to the first condition variable of the map update process satisfying the process wake-up condition, waking up the map update process in the first thread, calling the current frame from the data cache area of ​​the map update process through the map update process and updating the first navigation map of the mobile device based on the current frame.

[0060] S230, in response to the condition variable of the loop detection process satisfying the process wake-up condition, wake up the loop detection process, call the current frame and multiple key frames from the key frame cache partition of the loop detection process through the loop detection process, determine the loop result of the current frame based on the multiple key frames and the current frame, and store it in the loop result cache partition of the data cache area.

[0061] S240, in response to the condition variable of the loop optimization process satisfying the process wake-up condition, waking up the loop optimization process, and determining the loop optimization results of multiple target frames between the current frame and the matching frame based on the loop results through the loop optimization process and storing them in the optimization record cache partition of the data cache area.

[0062] S250, in response to the second condition variable of the map update process satisfying the process wake-up condition, waking up the map update process in the second thread, and determining a second navigation map of the mobile device based on loop optimization results of multiple target frames through the map update process and replacing the first navigation map.

[0063] Based on the above Figure 2It can be seen that there are two relatively independent processing processes in process P200, specifically including the regular map update process formed by S210 and S220 and the loop optimization process formed by S210, S230 to S250. Among them, relatively independent can mean that the regular map update process and the loop optimization process can run in parallel in different threads. In particular, the map update process involved in the regular map update process runs in the first thread, and the map update process involved in the loop optimization process runs in the second thread.

[0064] In the aforementioned S210, during the operation of the self-mobile device, the sensor integrated in the self-mobile device (such as the aforementioned sensor 130) can periodically collect sensor data, and encapsulate the sensor data at each moment into a data frame, and the data frame corresponding to the current moment can be recorded as the current frame. Based on the aforementioned content, it can be known that each data frame can include point cloud data and posture data at the corresponding moment.

[0065] In some embodiments, the sensor may not be able to directly obtain point cloud data and posture data during actual application, and the corresponding sensor data may be first obtained and processed to determine the point cloud data and posture data. Optionally, the point cloud data and posture data determination algorithm may also be integrated into the data acquisition process or configured as an independent process.

[0066] For example, the point cloud data determination algorithm may include a coordinate conversion algorithm for the point cloud data, that is, the sensor data acquired by the sensor is often characterized as depth information based on the sensor coordinate system. When determining the point cloud data, the depth data may be subjected to a coordinate conversion to determine the depth information in the coordinate system of the self-mobile device and use it as the point cloud data. Optionally, the aforementioned determination algorithm may also include other related algorithms for depth data (such as conventional algorithms such as filtering and smoothing).

[0067] For another example, the algorithm for determining the posture data may include a mileage calculation method. That is, the sensor data corresponding to the posture data is often configured as multi-axis inertial data (such as acceleration in three directions and angular velocity in three directions). When determining the posture data, the mileage calculation method can be used to integrate the acquisition interval between two data frames to determine the posture change between the two data frames, and then the posture data at the previous moment is combined to determine the posture data at the current moment.

[0068] In some embodiments, when caching the current frame, it can be saved in different regions based on the frame type. That is, the data cache area of ​​each process is configured with a normal frame cache partition and a key frame cache partition, so that partitions are saved based on the frame type of the current frame. In addition, it should be noted that in order to distinguish each data frame, the data frame can be configured with identification information (such as a global ID, timestamp, etc.). The subsequent data update determination can also be determined based on whether the identification information of the latest data in the corresponding queue of the data cache area is not processed.

[0069] In addition, as mentioned above Figure 1 As shown, each process may be configured with a data cache area. Then, in the aforementioned S210, the current frame may be copied one by one to each data partition of the data cache area of ​​each process. For more information about frame type determination and partition storage, please refer to Figure 3 and its related description.

[0070] In some embodiments, when executing the aforementioned S210, when data is updated in the data cache area of ​​each process, the corresponding process can be triggered based on the data update situation.

[0071] As mentioned above, the first condition variable of the map update process can monitor the data update status of the key frame buffer partition and the common frame buffer partition of the map update program, so as to be configured as a valid value when the data is updated to control the map update process to wake up in the first thread to execute the above S220.

[0072] As mentioned above, the map update process may have a first thread and a second thread. The map update processes of different threads may be called based on the same map update program, so as to generate different navigation maps according to different scenarios. Among them, the map update process of the first thread generally resides in the first thread, so as to update the corresponding navigation map based on the update of the data frame. Among them, the navigation map determined by the map update process of the first thread may be recorded as the first navigation map.

[0073] In some embodiments, the specific update method of the map update process can be configured based on actual needs. Figure 1 Generally, it is configured as a grid map. In the map update process, the occupancy status of each grid in the grid map can be determined based on the point cloud data to update the grid map.

[0074] Taking into account that the self-mobile device generally only involves two-dimensional information when navigating, the aforementioned grid map can be configured as a two-dimensional grid map. Therefore, when determining the grid map, the point cloud data can be first filtered and projected based on the self-mobile device, and only the point cloud data related to the movement of the self-mobile device is retained (such as removing higher point cloud data based on the height of the self-mobile device), and projected into a two-dimensional plane. Then extract the nearest point in the two-dimensional plane to convert the three-dimensional point cloud data into two-dimensional point cloud data. After completing the data conversion, the corresponding map update algorithm (such as the Bresenham algorithm) can be called based on the posture data of the corresponding data frame to update the grid map to determine the first navigation map perceived at the current moment.

[0075] In some embodiments, the aforementioned grid map may also be configured as a grid map in a three-dimensional space. In this case, the relevant algorithm may be an update algorithm for a three-dimensional grid map (such as a ray projection algorithm), which will not be elaborated herein.

[0076] In some embodiments, considering that the subsequent loop optimization process is performed based on key frames to optimize the posture data of the key frames, in order to match the optimized posture data, the map can also be updated based on the posture data of the key frames during the map update process.

[0077] In some embodiments, when updating the navigation map through the map update process, the normal frame can be bound to the key frame first. Considering the timing of data frame acquisition, the key frame bound to each normal frame can be the key frame that is before the normal frame and has the closest time distance to the normal frame. Therefore, when performing the first navigation map update:

[0078] For ordinary frames: in response to the current frame being an ordinary frame, determine the binding key frame corresponding to the current frame and determine the relative posture of the current frame and the binding key frame, and update the first navigation map based on the relative posture of the current frame and the binding key frame and the point cloud data of the current frame.

[0079] For a key frame: in response to the current frame being a key frame, updating the first navigation map based on the point cloud data and the pose data of the current frame.

[0080] It should be noted that although the above only involves the relative pose of the ordinary frame and the key frame, the actual update can also be calculated based on the relative pose to determine the actual pose of the ordinary frame. The above expression is mainly convenient for replacing the pose data of the key frame after the pose data is changed, that is, after the pose data of the key frame changes, the actual pose of the ordinary frame can be adjusted based on the pose change and the relative pose of the two, without the need for separate conversion, thereby improving the processing rate.

[0081] The aforementioned S230 to S250 is a posture optimization process based on loop frames, wherein the loop frames can specifically refer to data frames of areas that the mobile device has passed through / perceived during its historical movement. That is, when a certain position is passed by a mobile device and then passes through the position again, the data frame corresponding to the moment of passing through the position again can be regarded as a loop frame, and the data frame corresponding to the moment of passing through the position in history can be recorded as a matching frame.

[0082] That is, when a loop frame appears, it means that the environment where the loop frame is located is perceived twice. Then, the actual posture error between the loop frame and the matching frame can be determined based on the point cloud data of the environment, thereby eliminating the cumulative error.

[0083] Considering that the posture of the self-moving device remains unchanged, it will generally continue to move in a specific direction and will not repeat a certain position. The loop frame often appears because the posture of the self-moving device changes (such as turning around, turning in a circle), etc. Therefore, in order to reduce the amount of calculation during loop detection, only the data frames with large posture changes can be detected. The above-mentioned ordinary frame can refer to the data frame with small posture changes of the self-moving device, and the key frame can refer to the data frame with large posture changes of the self-moving device.

[0084] In particular, the loop detection process in the present application can be triggered in response to a new key frame, that is, the conditional variable of the loop detection process can monitor the data update of the key frame cache partition of the loop detection process, so that when the key frame cache partition data is updated, the conditional variable is in a valid state to trigger the loop detection process to execute the aforementioned S230.

[0085] In S230, when the current frame is a key frame, whether the current frame is a loop frame can be determined through a loop detection process, so that when the current frame is a loop frame, a matching frame of the current frame and a relative posture between the current frame and the matching frame can be determined.

[0086] In the aforementioned S230, the determination of the loop frame is generally based on the candidate frame determined based on the posture data, and then the matching frame is determined based on the point cloud data of the candidate frame. Exemplarily, the posture data can be accumulated in reverse with the current frame as the starting point to determine multiple candidate frames whose position integrals of the posture data are close to the current frame, and then the point cloud data of the candidate frame is aligned with the point cloud data of the current frame to determine the matching frame. Among them, when the aforementioned process cannot find a matching frame, it can be determined that the current frame is not a loop frame, thereby terminating the loop optimization process of the current frame and performing detection of the next data frame. For more information on determining loop frames, please refer to Figure 4 and its related description.

[0087] In some embodiments, if the current frame is a loop frame, the loop result of the current frame can be stored in the loop result cache partition of the data cache space of the subsequent process. The loop result is generally stored in the loop result cache partition of the subsequent loop optimization process.

[0088] As mentioned above, the condition variable of the loop optimization process can monitor the data update status of the loop result cache partition of the loop optimization process, so that when the loop result cache partition data is updated (i.e., a loop data frame appears), the condition variable is in a valid state to trigger the loop optimization process to execute the above S240.

[0089] In the aforementioned S240, the loop closure optimization process can call the loop closure result and correct the posture data of multiple target frames between the current frame and the matching frame based on the loop closure result to determine the corrected posture data of each target frame. Among them, the correction process can be performed based on a common posture correction algorithm (such as image optimization, factor graph method, etc.). Specifically, the relative posture of the current frame and the matching frame is often configured as a constraint condition, so as to execute the corresponding posture correction algorithm to determine the corrected posture data of each key frame.

[0090] In some embodiments, to reduce the amount of computation, the loop closure optimization process can be constructed based on the incremental principle, that is, when a loop closure frame is detected, only the key frames between the loop closure frame and its matching frame can be corrected, rather than a global correction. For more information about incremental nonlinear optimization, see Figure 4 and its related description.

[0091] In some embodiments, considering that the aforementioned correction process can update the pose data of the key frame, the corrected pose data can be synchronized to each data cache area after determining the corrected pose data. In order to facilitate triggering subsequent processing, the correction process can be configured as an optimization record (which can include the modified key frame ID) stored in the optimization record cache partition of the data cache area. Among them, the optimization record is generally stored in the optimization record cache partition of the map update process.

[0092] As mentioned above, the second condition variable of the map update process can monitor the data update status of the optimization record cache partition of the map update program, so that when the optimization record cache partition data is updated (i.e., a new optimization record appears), the second condition variable is in a valid state, thereby triggering the map update process in the second thread to execute the aforementioned S250.

[0093] The second thread may refer to a thread / thread combination in the processor that is different from the aforementioned first thread. That is, the map update processes of different threads can be executed in parallel, so that the navigation map optimization based on the loop frame can be achieved without affecting the real-time update of the navigation map. Among them, the map update process of the first thread generally runs in the foreground, so as to directly present the first navigation map. The map update process of the second thread generally runs in the background, and after it completes the optimization, the second navigation map it determines replaces the aforementioned first navigation map.

[0094] The map update process of the second thread can be similar to the map update program of the first thread, so as to perform map modeling based on the corrected posture data to determine the second navigation map. Considering that the corrected posture data can be directly synchronized to each data cache area in the aforementioned steps, the map update process of the second thread can directly call each key frame to construct the navigation map to determine the second navigation map, and replace the first navigation map with the second navigation map after the reconstruction is completed.

[0095] In some embodiments, considering that the aforementioned loop optimization process can be configured as an incremental process, the map update process of the second thread can also match the process and be configured as an incremental map update process. Combined with the aforementioned keyframe-based navigation map update process, the incremental navigation map update process can be: first obtain the historical navigation map at the acquisition time of the matching frame from the data cache area of ​​the map update process. Then determine the multiple bound ordinary frames bound to the target frame and the relative posture of the bound ordinary frames relative to the target frame. Finally, based on the corrected posture data of the multiple target frames and the relative posture of the bound ordinary frames and the target frame, update the historical navigation map to determine the second navigation map.

[0096] In addition, considering the first navigation point Figure 1 Generally, the data cache area of ​​the map update process may not be able to cache the data. In this case, the aforementioned process may also directly construct a navigation map based on the corrected key frames and ordinary frames to determine the second navigation map.

[0097] In some embodiments, considering the running time of the second thread, multiple data frames may be obtained from the mobile device in the process of actually determining the second navigation map, then the map update program of the second thread can update the second navigation map according to the newly obtained data frames until the second navigation map is synchronized with the first navigation map. For details, see Figure 5 and its related description.

[0098] Therefore, based on the above, a multi-threaded map update method and its storage medium, electronic device and program product are provided. The present application achieves the decoupling of the data relationship of each link in the map update method by caching data outside the process, and adjusts the conditional variables of the process by updating the cached data to achieve the decoupling of the trigger logic between each link. Therefore, the present application can split the map update method into multiple independently running processes, and in actual execution, each process can be executed in a different thread, thereby realizing multi-linear parallel update of the map to improve the execution speed of map update. In particular, in order to eliminate the accumulated errors in the map, it can be determined whether the current frame collected at the current moment is a loop frame. When the current frame is a loop frame, the map correction based on the loop frame and the map update based on the current frame can be implemented by calling the same link in parallel in different threads, so that the elimination of the accumulated errors in the map does not affect the normal display of the map.

[0099] Exemplary Data Caching Method

[0100] To further illustrate the aforementioned frame data caching process, the present application also provides an exemplary flow chart of a data caching method ( Figure 3 ).

[0101] like Figure 3 As shown, process P300 may include the following steps:

[0102] S310, determining a posture change of the mobile device at the current moment based on the posture data of the current frame.

[0103] S320: Determine the frame type of the current frame based on the posture change.

[0104] S330 . Copy the current frame to a corresponding partition in the data cache area of ​​the data acquisition process based on the frame type of the current frame.

[0105] S340 , in response to data update in the data cache area of ​​the data acquisition process, copy the current frame to the data cache area of ​​other processes.

[0106] In the aforementioned S310, the posture change of the self-moving device at the current moment can generally be determined by the time series integration of the inertial sensor data. For example, the posture data of the frame data can be represented by the time series inertial sensor data of the frame data between the previous moment and the current moment, and the posture change of the self-moving device between the two times can be determined by the mileage calculation method.

[0107] As mentioned above, the frame type can generally be configured based on a threshold value. When the posture change is less than the threshold value, the frame type of the current frame is configured as a normal frame; when the posture change is greater than the threshold value, the frame type of the current frame is configured as a key frame.

[0108] Considering that subsequent processing may involve simultaneous processing of common frames and key frames and needs to rely on specific data of key frames, after determining the frame type, the frame type identifier can be added to the data structure of the data frame.

[0109] In some embodiments, considering the binding of the aforementioned common frame and key frame, the representation form of the posture data in each frame data can be adjusted. Among them, the posture data of the key frame can reflect its actual position and posture, and the posture data of the common frame can reflect its relative position (also known as position change) and relative posture (also known as posture change) relative to the bound key frame.

[0110] In the cache data, the frame data can be cached through double buffering, that is, the aforementioned data acquisition process can be configured with a data cache area, and the subsequent process can also be configured with a data cache area. Thus, each data frame is stored in each data cache area. As a result, when the subsequent process calls data, it will only access its own data cache area, and will not access the data cache area of ​​the data acquisition process, which reduces the data call between threads, thereby greatly reducing thread blocking.

[0111] In order to further avoid read-write conflicts, especially read-write conflicts in the data cache area of ​​the data acquisition process. When copying the data in the data cache area of ​​the data acquisition process to the data cache area of ​​each process, it can be executed based on the data update status and read-write status of the data cache area of ​​the data acquisition process. Considering that this step is executed by the data acquisition process, its execution process does not require conditional variables and can be executed by directly reading the corresponding status.

[0112] Furthermore, considering that there may be a read-write conflict during data replication, the aforementioned data cache area may be configured with an occupied flag, and the aforementioned S340 may further include the following steps:

[0113] S341. Adjust the occupation flag of the data cache area of ​​the data acquisition process based on the writing status of the data cache area of ​​the data acquisition process.

[0114] S342: In response to the change of the occupation flag and the occupation flag being in an idle state, determine that the data in the data cache area is updated and copy the current frame to the data cache area of ​​other processes.

[0115] The occupation flag can reflect the read and write status of the current data cache area. The occupation flag includes a write state and an idle state. To avoid simultaneous data read requests and data write requests for the data cache area, the data cache area cannot be read when the occupation flag is in a write state.

[0116] In addition, it should be noted that the writing situation in the aforementioned S341 may include data being written to the disk during the data acquisition process, and may also include the modification of the posture data in the subsequent links. Therefore, based on the aforementioned S341, when the occupancy mark is adjusted, it can be determined that new data (such as the current frame or corrected posture data) is written to the data, so that the corresponding partition data can be determined to be updated to trigger the subsequent steps.

[0117] It should be noted that, considering that the data cache area of ​​other processes may also have read-write conflicts, an occupation flag may also be configured for it based on the above process.

[0118] Exemplary loop correction process

[0119] To further illustrate the loop correction process provided by the present application, the present application also provides a signaling diagram of the loop correction process ( Figure 4 ). Figure 4 The signaling diagram shown can reflect the transmission relationship between data and signals among three parallel processes. Figure 4 The cross-thread trigger in can be understood as a process trigger based on conditional variables.

[0120] Combining the above content with Figure 4 It can be seen from the process P400 shown that when the loop detection process is triggered by a key frame, S411 to S415 in P400 can be executed to determine the loop result and trigger the loop optimization process, thereby executing S421 to S424 in P400, thereby determining the loop optimization result and triggering the map update process of the second thread, thereby executing S431 to S433 in P400.

[0121] That is Figure 4 As shown, in P400, the loop detection process can execute the following steps in response to the newly added key frame:

[0122] S411. Determine a posture search tree for each key frame based on the posture data of multiple key frames.

[0123] S412: Match the posture data of the current frame with the posture search tree of each key frame, and determine at least one candidate frame from the multiple key frames.

[0124] S413: Determine a plurality of three-dimensional networks of different resolutions for at least one candidate frame.

[0125] S414: performing point cloud registration on each three-dimensional network based on the point cloud data of the current frame to determine the registration result of the current frame.

[0126] S415. In response to the registration result including the target candidate frame in which each three-dimensional network matches the current frame, the current frame is determined to be a loop frame, the target candidate frame is a matching frame, and a relative pose between the current frame and the matching frame is determined.

[0127] Based on the above process, the loop closure optimization process can perform the following steps in response to the loop closure result:

[0128] S421. Call multiple target frames from the key frame cache partition.

[0129] S422: Determine multiple relative factor graphs based on multiple target frames.

[0130] S423: Determine a loop closure factor graph based on the relative pose between the current frame and the matching frame.

[0131] S424: Perform nonlinear optimization on the pose data of multiple target frames based on multiple relative factor graphs and loop closure factor graphs to determine loop closure optimization results.

[0132] Based on the above process, the map update process of the second thread may perform the following steps in response to the loop optimization result:

[0133] S431. Call multiple target common frames from the common frame buffer partition of the map update process based on the loop optimization result.

[0134] S432. Determine the corrected posture data of the target ordinary frame based on the binding relationship between the target ordinary frame and the target frame and the corrected posture data of the target frame.

[0135] S433, determining a second navigation map based on the corrected posture data of multiple target frames and the corrected posture data of multiple target ordinary frames.

[0136] The aforementioned S411 to S412 can be understood as a process of determining a candidate frame of the current frame from a key frame. Specifically, in the aforementioned S411 to S412, the determination of the candidate frame can be achieved through a posture search tree. The posture search tree can be a tree-like search structure constructed based on posture data.

[0137] Exemplarily, the posture search tree can be a KDTree (K-dimensional binary search tree) constructed based on posture data (such as three degrees of freedom positions). Among them, the current frame and multiple historical key frames can be called from the key frame cache partition first, so as to construct a KDTree based on the posture data of the historical key frames. Then all key frames that are close to the current frame are found in the KDTree. Among them, the construction and search of KDTree can refer to the description of the relevant technology, which will not be repeated here.

[0138] In some embodiments, in order to prevent the current frame from finding the place it just walked through and being defined as a loop frame, the identification information of the candidate frame determined above can satisfy a preset interval relationship with the identification information of the current frame. For example, the ID (or timestamp) of the candidate frame can be separated from the corresponding identification information of the current frame by at least a preset number of data frames (or a preset time interval).

[0139] After determining a plurality of candidate frames, S413 to S415 may be executed to determine whether the current frame is a loop frame, wherein if the current frame is not a loop frame, the current process is terminated directly, and if it is a loop frame, the aforementioned loop result is determined.

[0140] Specifically, after obtaining the candidate frame, the point cloud information of several key frames near the candidate frame is combined to construct a local map. The local map is then point cloud registered with the 3D point cloud of the current frame. Considering the large amount of calculation required for point cloud configuration, the local map can be sampled at different resolutions to determine 3D networks of different resolutions, and then point cloud registration of the 3D networks of each resolution with the point cloud data of the current frame is performed.

[0141] The above configuration process can be performed step by step based on low resolution, and each local map that cannot be registered is eliminated during the registration process, so as to determine the target candidate frame in which each 3D network matches the current frame. At this time, the current frame can be determined to be a loop frame, and the target candidate frame is a matching frame.

[0142] Considering that the registration process of point cloud data and the aforementioned three-dimensional network often requires spatial movement, the relative posture of the current frame and the matching frame can be determined by the registration relationship between the current frame point cloud data and the high-resolution three-dimensional network of the target candidate frame and the position of the mobile device in the three-dimensional network.

[0143] From the above S421 to S424, it can be known that the above loop process can be characterized as an incremental factor graph method.

[0144] In the above S421 to S424, the factor graph method is an important optimization method. Specifically, the factor graph method represents and processes data by constructing an undirected graph model consisting of variable nodes and factor nodes. The variable nodes represent the state parameters of the system, such as posture transformation; the factor nodes represent observation constraints or measurement functions. Based on the factor graph, the state parameters (such as posture data) can be optimized.

[0145] The aforementioned incremental factor graph method can be understood as a method of introducing a loop closure factor graph into conventional constraints. Specifically, the incremental factor graph method can be performed on the key frame (referred to as the target frame) between the loop closure frame and the matching frame. The constraints of the loop closure frame are introduced into the general factor graph constraints of the target frame, thereby performing nonlinear optimization.

[0146] Based on the aforementioned incremental factor graph method, multiple relative factor graphs can be determined based on multiple target frames. Then, a loop closure factor graph can be determined based on the relative pose between the current frame and the matching frame. Finally, the pose data of multiple target frames are nonlinearly optimized based on the multiple relative factor graphs and the loop closure factor graph to determine the corrected pose data of multiple target frames. The relative factor graph reflects the relative pose between two consecutive target frames of the self-mobile device, and the loop closure factor graph can reflect the relative pose between the current frame (loop closure frame) and the matching frame.

[0147] Specifically, in the loop closure optimization process, the aforementioned relative factor graph can be determined as the key frame cache partition is updated. That is, the loop closure optimization process can determine the relative pose between the newly added key frame and the previous key frame as the key frame cache partition is updated, thereby constructing the relative factor graph.

[0148] The aforementioned loop closure factor graph can be determined based on the update of the cache area of ​​the aforementioned loop closure result. That is, the loop closure result (i.e., the loop closure frame, the matching frame of the loop closure frame, and the relative pose between the two) can be obtained from the cache area. Thus, a new real constraint can be generated based on the relative position and added to the factor graph (referred to as the loop closure factor graph).

[0149] Different from the above process, in the traditional factor graph, it is often necessary to add a strong prior factor in the first frame, so as to globally optimize during loop closure optimization to eliminate the above prior factor. In this application, the relationship between key frames can be reflected by the relative factor graph, and the real corresponding relationship can be reflected by the loop closure factor graph, so there is no need to reconstruct the factor graph with the prior factor and the head, thus realizing an incremental factor graph method, which can reduce the computational overhead while eliminating the cumulative error.

[0150] Based on the aforementioned updated factor graph, incremental nonlinear optimization can be performed based on the factor graph to determine the optimized corrected pose data of the target frame between the loop frame (current frame) and the matching frame.

[0151] Conventional nonlinear optimization methods require global optimization of all variables in the entire system each time new data is added. However, the incremental nonlinear optimization method provided by the present application optimizes only the local variables affected by the new data each time new data is added, rather than globally recalculating, thus greatly improving the computational efficiency. Specifically, for larger-scale maps, the optimization can be completed in just tens of milliseconds using incremental nonlinear optimization.

[0152] According to the above S431 to S433, when actually updating the map, the map update process of the second thread is similar to that of the first thread. The specific difference is that the map update process of the second thread often needs to update multiple data frames at a time, so in actual application, each data frame can be updated one by one.

[0153] Considering that the map update process of the second thread has a large amount of data, it often takes a certain amount of time to process. After the loop frame appears, multiple loop frames often appear continuously. At this time, the map update process of the second thread can be adjusted according to the actual situation.

[0154] In some embodiments, if other loop frames appear before the map update process of the thread completes the update, the current map update may be terminated, so that the map is updated based on the latest loop frame.

[0155] In some embodiments, in order to avoid the situation where repeated termination of the map update process of the second thread may eventually lead to failure in correction, a map update process may be triggered in the third thread when a new loop frame appears. The map update program of the second thread will not be interrupted, and the map update program of the third thread may be re-triggered in response to the new loop frame. This ensures that at least the navigation map can be corrected based on the map update program of the second thread.

[0156] Exemplary first navigation map correction method

[0157] In some embodiments, to further illustrate the process of replacing the first navigation map with the second navigation map, the present application also provides an exemplary flow chart of a method for correcting the first navigation map ( Figure 5 ).

[0158] like Figure 5 As shown, process P500 may include the following steps:

[0159] S510: Initialize and call a map update process in the second thread to determine a background map update process.

[0160] S520: Determine a second navigation map based on the corrected position data of the plurality of target frames through a background map update process.

[0161] S530: Determine whether the second navigation map is synchronized with the first navigation map.

[0162] S540: Replace the first navigation map with the second navigation map, and terminate the background map update process.

[0163] S550: Retrieve a difference data frame between the first navigation map and the second navigation map from the data cache area, and update the second navigation map based on the difference data frame.

[0164] S540 is the "yes branch" of S530, and S550 is the "no branch" of S530. In particular, after S550 is executed, S530 may be executed again to form a loop processing process until the second navigation map is synchronized with the first navigation map.

[0165] In some embodiments, whether the second navigation map is synchronized with the first navigation map can be determined based on whether the latest data frames in each navigation map are the same. If the two are not the same, it means that the first navigation map received a new data frame during the update process of the second navigation map. To ensure the continuous use of the map, it is necessary to execute the above S550 to update the missing difference data of the second navigation map.

[0166] Exemplary Devices

[0167] Combination of the above Figures 1 to 5 , describes the method embodiment of the present application in detail, and the device embodiment of the present application is described in detail below. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the part not described in detail can refer to the previous method embodiment.

[0168] The present application also provides a multi-threaded map update system, including a functional module for implementing the multi-threaded map update method provided by the present application. Among them, a three-dimensional map update system can be integrated in the operating environment of an electronic device, thereby providing a three-dimensional map update function for the electronic device.

[0169] For example Figure 6 As shown, Figure 6 3D map updating system 600 may include a data acquisition module 610 , a first map updating module 620 , a loop detection module 630 , a loop optimization module 640 and a second map updating module 650 .

[0170] The data acquisition module 610 can be used to obtain the current frame at the current moment through the data acquisition process, and copy the current frame to the corresponding data cache area based on the frame type of the current frame, wherein the current frame includes the point cloud data and posture data at the current moment, the frame type includes key frames and ordinary frames, and the data cache area includes ordinary frame cache partitions and key frame cache partitions.

[0171] The first map update module 620 can be used to wake up the map update process in the first thread in response to the first condition variable of the map update process satisfying the process wake-up condition, call the current frame from the data cache area of ​​the map update process through the map update process, and update the first navigation map of the mobile device based on the current frame, wherein the first condition variable of the map update process reflects the data update status of the key frame cache partition and the ordinary frame cache partition of the map update program.

[0172] The loop detection module 630 can be used to wake up the loop detection process in response to the conditional variables of the loop detection process satisfying the process wake-up conditions, call the current frame and multiple key frames from the key frame cache partition of the loop detection process through the loop detection process, determine the loop result of the current frame based on the multiple key frames and the current frame, and store it in the loop result cache partition of the data cache area, wherein, when the current frame is a loop frame, the loop result includes the current frame, the matching frame corresponding to the current frame, and the relative posture between the current frame and the matching frame, and the conditional variables of the loop detection process reflect the data update status of the key frame cache partition of the loop detection process.

[0173] The loop optimization module 640 can be used to wake up the loop optimization process in response to the conditional variables of the loop optimization process satisfying the process wake-up conditions, and determine the loop optimization results of multiple target frames between the current frame and the matching frame based on the loop results through the loop optimization process and store them in the optimization record cache partition of the data cache area, wherein the conditional variables of the loop optimization process reflect the data update status of the loop result cache partition of the loop optimization process, and the loop optimization results include corrected pose data.

[0174] The second map update module 650 can be used to wake up the map update process in the second thread in response to the second condition variable of the map update process satisfying the process wake-up condition, and determine the second navigation map of the mobile device based on the loop optimization results of multiple target frames through the map update process and replace the first navigation map, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

[0175] The present application also provides an electronic device, wherein the electronic device can be specifically configured as the aforementioned Figure 1 In a practical application, the electronic device can be used as a user terminal to implement the multi-threaded map update method provided in the present application. For example, a sensor on the electronic device can determine the data frame (including posture data and point cloud data) at each moment and update the first navigation map. When the frame data at the current moment is a loop frame, the electronic device can correct the posture data, thereby determining a second navigation map based on the corrected posture data to replace the first navigation map.

[0176] The present application also provides a computer program product, including a computer program / instruction. When the computer program / instruction processor in the computer program product provided by the present application is executed, the multi-threaded map updating method provided by the present application can be implemented.

[0177] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0178] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0179] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, operation and maintenance management (operation administration and maintenance, OAM) or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-threaded map updating method, characterized in that: The multi-threaded map updating method is applied to a mobile device, comprising: Acquire a current frame at a current moment through a data acquisition process, and copy the current frame to a corresponding data cache area based on a frame type of the current frame, wherein the current frame includes point cloud data and pose data at the current moment, the frame type includes a key frame and a common frame, and the data cache area includes a common frame cache partition and a key frame cache partition; In response to a first condition variable of a map update process satisfying a process wake-up condition, waking up the map update process in a first thread, calling the current frame from a data cache area of ​​the map update process through the map update process, and updating a first navigation map of the mobile device based on the current frame, wherein the first condition variable of the map update process reflects data update conditions of a key frame cache partition and a common frame cache partition of the map update program; In response to a condition variable of a loop detection process satisfying a process wake-up condition, the loop detection process is woken up, the current frame and multiple key frames are called from a key frame cache partition of the loop detection process through the loop detection process, a loop result of the current frame is determined based on the multiple key frames and the current frame, and stored in a loop result cache partition of the data cache area, wherein when the current frame is a loop frame, the loop result includes the current frame, a matching frame corresponding to the current frame, and a relative position between the current frame and the matching frame, and the condition variable of the loop detection process reflects the data update status of the key frame cache partition of the loop detection process; In response to a condition variable of the loop closure optimization process satisfying a process wake-up condition, waking up the loop closure optimization process, determining loop closure optimization results of multiple target frames between the current frame and the matching frame based on the loop closure result through the loop closure optimization process and storing them in the optimization record cache partition of the data cache area, wherein the condition variable of the loop closure optimization process reflects the data update status of the loop closure result cache partition of the loop closure optimization process, and the loop closure optimization result includes the corrected pose data; In response to a second condition variable of a map update process satisfying a process wake-up condition, the map update process is woken up in a second thread, and a second navigation map of the mobile device is determined by the map update process based on loop optimization results of the multiple target frames and the first navigation map is replaced, wherein the second condition variable of the map update process reflects the data update status of the optimized record cache partition of the map update program.

2. The multi-threaded map updating method according to claim 1, characterized in that: The step of copying the current frame to a corresponding data buffer area based on the frame type of the current frame includes: Determining a posture change of the mobile device at a current moment based on the posture data of the current frame; Determining a frame type of the current frame based on the posture change, wherein when the posture change is less than a threshold, the frame type of the current frame is configured as a normal frame; when the posture change is greater than the threshold, the frame type of the current frame is configured as a key frame; Copying the current frame to a corresponding partition in a data cache area of ​​the data acquisition process based on a frame type of the current frame; In response to data update in the data cache area of ​​the data acquisition process, the current frame is copied to the data cache area of ​​other processes.

3. The multi-threaded map updating method according to claim 2, characterized in that: In response to the data update of the data cache area, copying the current frame to the data cache area of ​​other processes includes: Adjusting an occupation flag of a data cache area of ​​the data acquisition process based on a write condition of the data cache area of ​​the data acquisition process, wherein the occupation flag includes a write state and an idle state; In response to the change of the occupancy flag and the occupancy flag being the idle state, it is determined that the data in the data cache area is updated and the current frame is copied to the data cache area of ​​other processes.

4. The multi-threaded map updating method according to claim 1, characterized in that: The determining, through the loop closure optimization process, loop closure optimization results of a plurality of target frames between the current frame and the matching frame based on the loop closure result includes: Recalling the plurality of target frames from the key frame buffer partition; Determine a plurality of relative factor graphs based on the plurality of target frames, wherein the relative factor graphs reflect the relative poses between two consecutive target frames of the self-moving device; Determine a loop closure factor graph based on a relative pose between the current frame and the matching frame; Based on the multiple relative factor graphs and the loop closure factor graphs, nonlinear optimization is performed on the pose data of the multiple target frames to determine the loop closure optimization result.

5. The multi-threaded map updating method according to claim 4, characterized in that: The determining of the second navigation map of the mobile device based on the loop optimization results of the plurality of target frames through the map update process includes: Based on the loop optimization result, calling a plurality of target ordinary frames from the ordinary frame buffer partition of the map update process, wherein the plurality of target ordinary frames include ordinary frames between the current frame and the matching frame; Determine the corrected pose data of the target ordinary frame based on the binding relationship between the target ordinary frame and the target frame and the corrected pose data of the target frame; A second navigation map is determined based on the corrected pose data of the multiple target frames and the corrected pose data of the multiple target ordinary frames.

6. The multi-threaded map updating method according to claim 1, characterized in that: Waking up the map update process in the second thread, determining the second navigation map of the mobile device based on the loop optimization results of the multiple target frames through the map update process and replacing the first navigation map, includes: Initializing a map update program of the map update process in the second thread, and determining the map update process of the second thread; Determining the second navigation map based on the corrected pose data of the plurality of target frames; Determining whether the second navigation map is synchronized with the first navigation map; If yes, replacing the first navigation map with the second navigation map, and terminating the map update process of the second thread; If not, a difference data frame is retrieved from the data cache area of ​​the map update process, and the second navigation map is updated based on the difference data frame, wherein the difference data frame reflects each data frame of the second navigation map that is not updated after the loop optimization.

7. The multi-threaded map updating method according to claim 1, characterized in that: The determining the loop result of the current frame based on the multiple key frames and the current frame includes: Determine a posture search tree for each key frame based on the posture data of the multiple key frames; Matching the posture data of the current frame with the posture search tree of each key frame, and determining at least one candidate frame from the multiple key frames; determining a plurality of three-dimensional networks of different resolutions for the at least one candidate frame; Performing point cloud registration on each three-dimensional network based on the point cloud data of the current frame to determine the registration result of the current frame; In response to the registration result including a target candidate frame whose three-dimensional networks all match the current frame, the current frame is determined to be a loop frame, the target candidate frame is the matching frame, and a relative pose between the current frame and the matching frame is determined.

8. The multi-threaded map updating method according to claim 1, characterized in that: The calling of the current frame from the data cache area of ​​the map update process through the map update process and updating the first navigation map of the mobile device based on the current frame comprises: In response to the current frame being a common frame, determining a bound key frame corresponding to the current frame and determining a relative position between the current frame and the bound key frame, and updating the first navigation map based on the relative position between the current frame and the bound key frame and point cloud data of the current frame; In response to the current frame being a key frame, the first navigation map is updated based on the point cloud data and posture data of the current frame.

9. The multi-threaded map updating method according to claim 8, characterized in that: The determining of a second navigation map from the mobile device based on loop optimization results of a plurality of target frames through a map update process includes: Determining a historical navigation map at the time of acquisition of a matching frame from a data cache area of ​​the map update process; Determine a plurality of bound common frames bound to the target frame and relative positions of the bound common frames relative to the target frame; The historical navigation map is updated based on the corrected posture data of multiple target frames and the relative posture between the bound common frame and the target frame to determine the second navigation map.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction processor is executed, a multi-threaded map updating method as described in any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the multi-threaded map updating method according to any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multi-threaded map updating method described in any one of claims 1-9.

Images