Visual map update processing method, mobile robot, and storage medium

Abstract

This application provides a visual map update processing method, a mobile robot, and a computer-readable storage medium. The method includes: determining a target sub-map associated with the robot's current processing keyframe from various sub-maps of a visual global map; filtering candidate keyframes within a preset range based on a first set of keyframes stored in the target sub-map and the prior pose of the current processing keyframe; selecting a preset number of keyframes based on the current task keyframe set and the historical task keyframe set, and matching them with the current processing keyframe to obtain matching keyframes for the current processing keyframe; optimizing the pose of the target sub-map based on the matching relationship between the matching keyframes and the current processing keyframe, and adding the current processing keyframe to the first keyframe set to obtain a second set of keyframes for the target sub-map. This application can improve the robot's positioning accuracy while ensuring that the visual global map can be updated according to environmental changes.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to a visual map update processing method, a mobile robot, and a computer-readable storage medium. Background Technology

[0002] Visual maps are the core foundational data for mobile robots to perform their tasks. The operating environments of some robots are dynamically changing. For example, the outdoor working environment of a lawnmower robot is significantly affected by the changing seasons, thus requiring continuous updates and optimization of the visual map based on the latest image data. However, current technologies for updating visual maps are time-consuming and slow, resulting in lower positioning accuracy for the robot. Summary of the Invention

[0003] This application provides a visual map update processing method, a mobile robot, and a computer-readable storage medium, which can improve the robot's positioning accuracy while ensuring that the visual global map can be updated with changes in the environment.

[0004] In a first aspect, this application provides a visual map update processing method, the method comprising: From the sub-maps of the visual global map, determine the target sub-map associated with the robot's current processing keyframe; Based on the first set of keyframes already stored in the target sub-map, and combined with the prior pose of the currently processed keyframe, candidate keyframes are filtered within a preset range. The candidate keyframes include the current job keyframe set and the historical job keyframe set. Based on the current job keyframe set and the historical job keyframe set, a preset number of keyframes are selected and matched with the current processing keyframe to obtain the matching keyframe of the current processing keyframe. Based on the matching relationship between the matching keyframe and the currently processed keyframe, the pose of the target sub-map is optimized, and the currently processed keyframe is added to the first keyframe set to obtain the second keyframe set of the target sub-map.

[0005] Secondly, this application also provides a mobile robot, which includes a camera, a processor and a memory, wherein the memory stores a computer program, and the processor executes any of the visual map update processing methods provided in this application when it calls the computer program in the memory.

[0006] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to execute the visual map update processing method described above.

[0007] In this application, firstly, by optimizing the pose of the target sub-map based on the matching relationship between the matched keyframes and the currently processed keyframes, and adding the currently processed keyframes to the first set of keyframes already stored in the target sub-map to obtain a second set of keyframes, the environmental information of the latest job-collected keyframes can be used to update the visual global map, ensuring that the visual global map can be updated with environmental changes and ensuring positioning accuracy. Secondly, by selecting a preset number of keyframes based on the current job keyframe set and the historical job keyframe set to match the currently processed keyframes, the number of matched keyframes can be limited, reducing the computational load of subsequent pose optimization and reducing computational blocking problems caused by too many matched keyframes, thereby reducing the time consumption of visual map update processing, improving the update speed of visual map, and improving the robot's positioning accuracy. Thirdly, by selecting a preset number of keyframes based on the current job keyframe set and the historical job keyframe set to match the currently processed keyframes, pose optimization can be performed using matched keyframes from different job cycles, thereby combining the latest and historical environments for pose optimization, improving the reliability of pose optimization, improving the accuracy of the target sub-map, and thus improving the robot's positioning accuracy to a certain extent. Therefore, this application can improve the robot's positioning accuracy while ensuring that the visual global map can be updated with changes in the environment. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic block diagram of the structure of a mobile robot provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the lawnmower robot provided in an embodiment of this application; Figure 3 This is a schematic flowchart of a visual map update processing method provided in an embodiment of this application; Figure 4 This is a schematic diagram of a robot operating in a bow-shaped pattern according to an embodiment of this application; Figure 5 It is aimed at Figure 4 The diagram shows an illustration of multiple sub-maps obtained by dividing the work area according to the work orientation. Figure 6 This is an illustrative diagram illustrating keyframe matching using keyframes from the first and second job cycles, respectively, provided in an embodiment of this application. Figure 7 This is a schematic flowchart of an embodiment of keyframe matching in this application. Figure 8 This is an illustrative diagram illustrating the updating of the visual map using keyframes collected in the first working cycle in an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0012] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0013] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0014] Visual maps are the core foundational data for mobile robots to carry out their operations. The operating environment of some robots is characterized by dynamic changes. For example, the outdoor operating scene of a lawnmower robot is affected by the changing seasons, resulting in significant environmental changes. Therefore, it is necessary to continuously update and optimize the visual map based on the latest image data.

[0015] In related technologies, visual map updates often involve directly merging multiple mapping data sets. However, this method causes the visual map size to increase continuously with each update, leading to a sustained rise in device memory and hard drive usage, which can easily cause memory and hard drive storage overload. Furthermore, updating the visual map is time-consuming and slow, resulting in lower positioning accuracy for the robot.

[0016] This application provides a visual map update processing method, a mobile robot, and a computer-readable storage medium. It can reduce the problem of continuously increasing memory and hard disk usage and memory or hard disk storage overload, while ensuring that the global visual map can be updated according to environmental changes, thereby improving the robot's positioning accuracy.

[0017] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0018] Figure 1 This is a schematic block diagram of the structure of a mobile robot provided in an embodiment of this application.

[0019] like Figure 1 As shown, the mobile robot 100 includes a processor 101, a memory 102, and a camera 103. The processor 101 and the memory 102 are connected via a bus, such as an I2C (Inter-integrated Circuit) bus. The mobile robot 100 can specifically be a lawnmower robot, a cleaning robot, a food delivery robot, etc. The mobile robot 100 can be configured with more or fewer components depending on the actual business scenario. Figure 2 As shown, Figure 2 This is a structural schematic diagram of a lawnmower robot provided in an embodiment of this application. Taking the mobile robot 100 as an example, the lawnmower robot can also be equipped with environmental sensors (such as lidar, infrared sensors, etc. in addition to cameras), anti-collision mechanisms, moving components, etc.

[0020] Camera 103 can be used as an environmental sensor to acquire image frames, such as the current processing key frame, the key frame of the first job cycle, and the key frame of the second job cycle.

[0021] Specifically, processor 101 provides computing and control capabilities to support the operation of the entire mobile robot 100. Processor 101 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0022] Specifically, the memory 102 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0023] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the mobile robot to which the embodiments of this application are applied. A specific mobile robot may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0024] The processor 101 is configured to run a computer program stored in the memory 102, and implement any of the visual map update processing methods provided in this application embodiment when executing the computer program. For example, the processor 101 is configured to run a computer program stored in the memory 102, and can implement the following steps when executing the computer program: From the sub-maps of the visual global map, determine the target sub-map associated with the robot's current processing keyframe; based on the first set of keyframes stored in the target sub-map, and combined with the prior pose of the current processing keyframe, filter candidate keyframes within a preset range, the candidate keyframes including the current operation keyframe set and the historical operation keyframe set; based on the current operation keyframe set and the historical operation keyframe set, select a preset number of keyframes to match with the current processing keyframe, obtain the matching keyframes of the current processing keyframe; based on the matching relationship between the matching keyframes and the current processing keyframe, optimize the pose of the target sub-map, and add the current processing keyframe to the first keyframe set, obtain the second set of keyframes of the target sub-map.

[0025] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: Obtain the prior pose of the current processing keyframe; based on the orientation angle range of each sub-map and the prior pose, determine the target sub-map associated with the current processing keyframe.

[0026] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: Obtain the prior pose of the currently processed keyframe; based on the prior pose, determine keyframes from the first keyframe set whose distance from the currently processed keyframe is within a preset distance range, as candidate keyframes; classify the candidate keyframes according to the operation cycle to obtain candidate keyframes for the first operation cycle and candidate keyframes for the second operation cycle, wherein the current operation keyframe set is the set of candidate keyframes for the first operation cycle, the historical operation keyframe set is the set of candidate keyframes for the second operation cycle, and the currently processed keyframe is the keyframe collected by the robot in the first operation cycle, the first operation cycle being lagging behind the second operation cycle.

[0027] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: Based on a preset number, the number of frames in the current job keyframe set, and the number of frames in the historical job keyframe set, the number of frames to be filtered in the first job cycle and the number of frames to be filtered in the second job cycle are determined. According to the number of frames to be filtered in the first job cycle, keyframes with higher feature similarity to the currently processed keyframes are selected from the current job keyframe set and used as the matching keyframes. According to the number of frames to be filtered in the second job cycle, keyframes with higher feature similarity to the currently processed keyframes are selected from the historical job keyframe set and used as the matching keyframes.

[0028] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: Obtain the global descriptor of the first candidate keyframe and the global descriptor of the currently processed keyframe, wherein the first candidate keyframe is a keyframe in the current job keyframe set; construct a first search structure based on the global descriptor of the first candidate keyframe; search the current job keyframe set based on the global descriptor of the currently processed keyframe and the first search structure to obtain N1 keyframes with the highest feature similarity to the currently processed keyframe, which are used as the matching keyframes, wherein N1 is equal to the number of screening frames in the first job cycle.

[0029] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: Obtain the global descriptor of the second candidate keyframe and the global descriptor of the currently processed keyframe, wherein the second candidate keyframe is a keyframe in the set of historical job keyframes; construct a second search structure based on the global descriptor of the second candidate keyframe; search the second candidate keyframe based on the global descriptor of the currently processed keyframe and the second search structure to obtain the N2 keyframes with the highest feature similarity to the currently processed keyframe, which are used as the matching keyframes, wherein N2 is equal to the number of screening frames in the second job cycle.

[0030] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: The system detects a first total number of frames in the current job keyframe set and a second total number of frames in the historical job keyframe set. If the first total number of frames is greater than or equal to half of a preset number, and the second total number of frames is greater than or equal to half of the preset number, then half of the preset number is used as the filter frame number for the first job cycle, and half of the preset number is used as the filter frame number for the second job cycle. Alternatively, if the first total number of frames is less than half of the preset number, then the first total number of frames is used as the filter frame number for the first job cycle, and the difference between the preset number and the first total number of frames is used as the filter frame number for the second job cycle. Alternatively, if the second total number of frames is less than half of the preset number, then the second total number of frames is used as the filter frame number for the second job cycle, and the difference between the preset number and the second total number of frames is used as the filter frame number for the first job cycle.

[0031] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: If the second keyframe set meets the preset grid update conditions, then the keyframes in each grid region of the target sub-map are simplified. The target sub-map corresponds to multiple grid regions. The preset grid update conditions include at least one of the following: the number of frames in the second keyframe set is greater than a preset frame number threshold, or the number of feature points in the second keyframe set is greater than a preset feature point number threshold.

[0032] In some embodiments, the processor 101 is configured to run a computer program stored in the memory 102, and when executing the computer program, it may perform the following steps: The target sub-map is divided into multiple grid regions on a preset coordinate plane according to a preset size. Based on the storage coordinates of each keyframe in the second keyframe set and the spatial range of each grid region, the grid region to which each keyframe belongs in the second keyframe set is determined. Based on the grid region to which each keyframe belongs in the second keyframe set, the keyframe with the latest acquisition timestamp is selected as the retained keyframe in each grid region. Other keyframes in each grid region except the retained keyframes are deleted to simplify the keyframes in each grid region of the target sub-map.

[0033] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the mobile robot described above can be referred to the corresponding process in the following embodiments of the visual map update processing method, and will not be repeated here.

[0034] The following will be based on Figure 1 Taking the mobile robot shown as the execution subject of the visual map update processing method as an example, this application provides a detailed description of the visual map update processing method provided in this embodiment. For simplicity and ease of description, the execution subject will be omitted in subsequent method embodiments. It should be noted that... Figure 1 The scenarios described are only used to explain the visual map update processing method provided in the embodiments of this application, but do not constitute a limitation on the application scenarios of the visual map update processing method provided in the embodiments of this application.

[0035] Please see Figure 3 , Figure 3 This is a schematic flowchart of a visual map update processing method provided in an embodiment of this application. It should be noted that, although in Figure 3 The logical order is shown in the flowcharts of other accompanying figures, but in some cases, the steps shown or described may be performed in a different order than that shown here. This visual map update processing method includes steps 301-304, wherein: 301. From the sub-maps of the visual global map, determine the target sub-map associated with the robot's current processing keyframe.

[0036] The visual global map is constructed by the robot based on image data of the work area collected by visual sensors (such as cameras). It includes environmental visual features, keyframe relationships, pose reference data, and sub-map division rules within the work area. Specifically, the visual global map can be a global map constructed based on visual SLAM (Simultaneous Localization and Mapping, a technique that uses visual sensors such as cameras to achieve simultaneous localization and mapping). The visual global map can be divided into multiple sub-maps. In some embodiments, the visual global map can be divided into multiple sub-maps according to geographical location. In some embodiments, to reduce keyframe matching time, the visual global map is divided into multiple sub-maps according to orientation, with each sub-map corresponding to a range of orientation angles.

[0037] To better understand the embodiments of this application, the following will first introduce some of the names involved in this embodiment: 1. Three-dimensional spatial coordinate system: A coordinate system used to describe the position of an object (such as a camera) in three-dimensional space. Specifically, a three-dimensional spatial coordinate system can be a world coordinate system; for example, a three-dimensional spatial coordinate system could be O... w X w Y w Z w .

[0038] 2. Camera coordinate system: This is a three-dimensional coordinate system with the origin at the camera's light source. For example, the camera coordinate system can be O... c X c Y c Z c .

[0039] 3. Camera pose: refers to the camera coordinate system (e.g., coordinate system O). c X c Y c Z c Relative three-dimensional coordinate system (such as O) w X w Y w Z w The position of (X) w ,Y w Z w ) and attitude (rX) w ,rY w ,rZ w ), that is (X w ,Y w Z w ,rX w ,rY w ,rZ w ), Xw Y w Z w rX w ,rY w 、rZ w Let X and Y represent the camera coordinate system in the three-dimensional spatial coordinate system, respectively. w Coordinates, camera coordinate system in three-dimensional spatial coordinate system Y w Coordinates, camera coordinate system in three-dimensional spatial coordinate system Z w Coordinates, camera coordinate system and OX w The angle between the axes, the camera coordinate system and the OY coordinate system w The angle between axes, camera coordinate system and OZ w The included angle of the axis.

[0040] Depending on the method of sub-map division, step 301 can be implemented in various ways, including, for example: (1) In some embodiments, the visual global map can be divided into multiple sub-maps according to geographical location, and each sub-map corresponds to a coordinate location range. In this case, step 301 may specifically include steps 3011A~3012A: 3011A. Obtain the prior pose of the current keyframe being processed.

[0041] Keyframes are images that meet specific criteria, such as time interval criteria (e.g., the frame interval from the previous keyframe is greater than a preset interval), spatial distance criteria (e.g., the displacement or rotation from the nearest keyframe exceeds a certain threshold), and tracking quality criteria (e.g., the number and distribution of feature points in the image meet requirements). For example, during robot operations, images with high information value (i.e., the number of feature points is greater than a preset threshold) and high stability (e.g., no obvious motion blur or exposure anomalies during acquisition) can be selected from continuously acquired ordinary images by the robot and used as keyframes.

[0042] The currently processed keyframe refers to the latest keyframe acquired during the robot's operation.

[0043] The pose of the current keyframe refers to the camera pose when acquiring the current keyframe, including its coordinate position and orientation. The coordinate position of the current keyframe refers to the camera coordinate system (e.g., coordinate system O) at the time of acquiring the current keyframe. c X c Y c Z c Relative three-dimensional coordinate system (such as O) w X w Y w Z w The position of (X) w ,Yw Z w The orientation of the current keyframe being processed refers to the camera coordinate system (e.g., coordinate system O) at the time the current keyframe was acquired. c X c Y c Z c Relative three-dimensional coordinate system (such as O) w X w Y w Z w The posture (rX) w ,rY w ,rZ w The orientation of the current processing keyframe directly reflects the robot's working direction when acquiring the current processing keyframe.

[0044] Among them, the prior pose of the current keyframe being processed refers to the camera pose corresponding to the current keyframe being processed, which is initially obtained by the Visual-Inertial Odometry (VIO) module carried by the robot.

[0045] In some embodiments, the robot can be equipped with a VIO module. During robot operation, the robot simultaneously acquires images through a camera and outputs the camera pose at the time of image acquisition using the visual localization algorithm of the VIO module (such as ORB-SLAM2, VINS, MSCKF, etc.). Thus, the robot can output the pose of the current keyframe (i.e., the camera pose at the time of acquiring the current keyframe) while simultaneously outputting the current keyframe being processed. In this case, the pose of the current keyframe, output during its acquisition, can be used as the prior pose of the current keyframe.

[0046] 3012A. Based on the spatial location range of each of the sub-maps and the prior pose, determine the target sub-map associated with the current processing keyframe.

[0047] The target submap refers to the submap associated with the currently processed keyframe.

[0048] For example, the pose of the current processing keyframe output by the robot during the acquisition of the current processing keyframe can be obtained as the prior pose of the current processing keyframe; then, based on the prior pose of the current processing keyframe, the coordinate position of the current processing keyframe is extracted; the coordinate position of the current processing keyframe is compared with the spatial position range of each sub-map to determine the sub-map corresponding to the spatial position range in which the coordinate position of the current processing keyframe falls, which is used as the target sub-map associated with the current processing keyframe.

[0049] For example, the visual global map is divided into sub-map A (Xw∈[0,10], Yw∈[0,10]) and sub-map B (Xw∈[10,20], Yw∈[0,10]) with a physical range of 10m×10m. If the coordinate position of the current processing keyframe is (8m, 5m, 0.2m), then the coordinate position of the current processing keyframe falls within the spatial position range of sub-map A, so sub-map A is determined as the target sub-map.

[0050] (2) In some embodiments, in order to reduce keyframe matching time, the visual global map is divided into multiple sub-maps according to the working orientation, and each sub-map corresponds to an orientation angle range. In this case, step 301 may specifically include steps 3011B~3012B: 3011B. Obtain the prior pose of the current keyframe being processed.

[0051] The implementation of step 3011B is similar to that of step 3011A. For details, please refer to the relevant explanations above. It will not be repeated here.

[0052] 3012B. Based on the orientation angle range of each of the sub-maps and the prior pose, determine the target sub-map associated with the current processing keyframe.

[0053] The orientation angle range of the sub-map can be defined according to the robot's working orientation. For example, ... Figure 4 and Figure 5 As shown, Figure 4 This is a schematic diagram of a robot operating in a bow-shaped pattern. Figure 5 It is aimed at Figure 4 The diagram illustrates the division of the work area into multiple sub-maps based on the work orientation. Assuming the lawnmower operates along a bow-shaped path, two sub-maps can be obtained based on the lawnmower's work orientation: Orientation 1 (e.g., assumed to be (rX)). w ,rY w ,rZ w =(0°,0°,0°)), Orientation 2 (e.g., assumed to be (rX) w ,rY w ,rZ w ) = (0°, 180°, 0°)) corresponds to sub-maps C and D respectively.

[0054] In some embodiments, the orientation corresponding to the earliest acquired keyframe within the sub-map can be used as the reference orientation of the sub-map, and the angle corresponding to the reference orientation of the sub-map can be directly used as the orientation angle range of the sub-map. For example, Figure 5 The orientation angle range of the neutron map C is (rX) w ,rY w ,rZ w= (0°, 0°, 0°), the orientation angle range of submap D is (rX) w ,rY w ,rZ w = (0°, 180°, 0°).

[0055] In some embodiments, the orientation corresponding to the earliest acquired keyframe within the sub-map can be used as the reference orientation of the sub-map. To reduce errors, an angle fluctuation range is set around each dimension of this reference orientation, serving as the orientation angle range of the sub-map. For example... Figure 5 The orientation angle ranges set for submap C are rXw∈[-5°,5°], rYw∈[-5°,5°], and rZw∈[-5°,5°], while the orientation angle ranges set for submap D are rXw∈[-5°,5°], rYw∈[175°,185°], and rZw∈[-5°,5°].

[0056] For example, the pose of the current keyframe output by the robot during keyframe acquisition can be obtained as the prior pose of the current keyframe; then, based on the prior pose of the current keyframe, the orientation of the current keyframe is extracted; the orientation of the current keyframe is compared with the orientation angle range of each sub-map to determine the sub-map corresponding to the orientation angle range d that the orientation of the current keyframe falls into, which is then used as the target sub-map associated with the current keyframe. Please refer to... Figure 5 In this way, the keyframes captured by the robot during its operation are divided and stored in different sub-maps according to the robot's working orientation (i.e., the orientation corresponding to the pose of the keyframe output by the robot when the keyframe is captured). For example, the lawnmower robot is divided according to... Figure 4 The robot travels in a bow-shaped pattern and performs lawn mowing. Keyframes captured during the robot's movement will be stored in different sub-maps based on the robot's orientation.

[0057] For example, the visual global map is divided into sub-map C (e.g., the orientation angle range of sub-map C is rXw∈[-5°,5°], rYw∈[-5°,5°], rZw∈[-5°,5°]) and sub-map D (e.g., the orientation angle range of sub-map D is rXw∈[-5°,5°], rYw∈[175°,185°], rZw∈[-5°,5°]) according to the operation orientation. If the orientation of the currently processed keyframe is (0°,172°,0°), then the orientation of the currently processed keyframe falls within the orientation angle range of sub-map D, so sub-map D is determined as the target sub-map.

[0058] 302. Based on the first set of keyframes already stored in the target sub-map, and combined with the prior pose of the currently processed keyframe, filter candidate keyframes within a preset range.

[0059] The candidate keyframes include the current job keyframe set and the historical job keyframe set.

[0060] The first keyframe set refers to the set of keyframes already stored in the target submap before the current keyframe is updated and added to the target submap.

[0061] There are several ways to implement step 302, including, for example: (1) In some embodiments, the historical job keyframe set and the current job keyframe set are respectively the set of the first candidate keyframes of the first job cycle and the set of the second candidate keyframes of the second job cycle. In this case, step 302 may specifically include the following steps 3021A~3023A: 3021A. Obtain the prior pose of the current processing keyframe.

[0062] The implementation of step 3021A is similar to that of step 3011A. For details, please refer to the relevant explanations above. It will not be repeated here.

[0063] Understandably, in the visual map update process using the current processing keyframe, the prior pose of the current processing keyframe can be stored in the database. When the corresponding step needs to use the prior pose, it can be read directly from the database without having to determine it repeatedly.

[0064] 3022A. Based on the prior pose, determine key frames from the first set of key frames whose distance from the currently processed key frame is within a preset distance range, and use them as candidate key frames.

[0065] The specific value of the preset distance range can be set according to the actual business scenario requirements. There is no restriction on the specific value of the preset distance range here. For example, the preset distance range can be less than or equal to the preset distance threshold.

[0066] Here, a candidate keyframe refers to a keyframe whose distance from the currently processed keyframe is within a preset distance range. For example, the distance between keyframe i in the first keyframe set and the currently processed keyframe can be determined as follows: based on the pose of keyframe i stored in the target sub-map, extract the coordinate position of keyframe i; based on the prior pose of the currently processed keyframe, extract the coordinate position of the current keyframe i; using the coordinate position of keyframe i and the coordinate position of the currently processed keyframe, calculate the distance between the keyframe and the currently processed keyframe.

[0067] For example, if the preset distance range is a spatial straight-line distance ≤ 5 meters, firstly, the coordinates of the current keyframe are extracted from its prior pose as (Xw1 = 8 meters, Yw1 = 5 meters, Zw1 = 0.2 meters). Then, the pose data of all keyframes stored in the first keyframe set of the target sub-map (e.g., {keyframe 1, keyframe 2, keyframe 3, ...}) are retrieved, and the coordinates of each keyframe are extracted. For example, the coordinates of keyframe 1 are (Xw2 = 7 meters, Yw2 = 6 meters, Zw2 = 0.2 meters), keyframe 2 are (Xw3 = 15 meters, Yw3 = 5 meters, Zw3 = 0.2 meters), and keyframe 3 are (Xw4 = 9 meters, Yw4 = 7 meters, Zw4 = 0.2 meters). Next, the spatial straight-line distance formula is used to calculate the spatial straight-line distance between each keyframe and the current keyframe. For example, the distance between keyframe 1 and the current keyframe is... meters, the distance of keyframe 2 is meters, the distance of keyframe 3 is Meters. The calculated straight-line distance in space is compared with the preset distance range (e.g., straight-line distance in space ≤ 5 meters). The distances of keyframe 1 and keyframe 3 meet the preset requirements, while the distance of keyframe 2 exceeds the preset range. Therefore, keyframe 1 and keyframe 3 are selected as candidate keyframes from the first set of keyframes, and so on.

[0068] In this way, by determining key frames from the first set of key frames that are within a preset distance range from the currently processed key frame based on prior pose, and using them as candidate key frames, distance filtering can be performed using prior pose, reducing the number of candidate key frames that need to be calculated for feature similarity in the subsequent process, thereby improving the key frame matching speed.

[0069] 3023A. The candidate keyframes are classified according to the job cycle to obtain the candidate keyframes for the first job cycle and the candidate keyframes for the second job cycle.

[0070] Wherein, the current task keyframe set is the set of candidate keyframes for the first task cycle, the historical task keyframe set is the set of candidate keyframes for the second task cycle, and the currently processed keyframe is the keyframe collected by the robot in the first task cycle, wherein the first task cycle lags behind the second task cycle.

[0071] The first job cycle is the job stage to which the current keyframe belongs, and the second job cycle is the historical job stage that is earlier than the first job cycle, that is, the first job cycle lags behind the second job cycle; each candidate keyframe carries the corresponding job cycle identifier information, which can be directly used to complete the classification.

[0072] For example, the job cycle identifier of each key frame in the candidate key frames can be retrieved, and the candidate key frames identified as the first job cycle can be classified into one category as the first candidate key frames. The set of the first candidate key frames forms the current job key frame set. The candidate key frames identified as the second job cycle can be classified into another category as the second candidate key frames. The set of the second candidate key frames forms the historical job key frame set.

[0073] Since the first and second operation cycles are two different operation cycles, by dividing the set of first candidate keyframes in the first operation cycle into the current operation keyframe set and the set of second candidate keyframes in the second operation cycle into the historical operation keyframe set, on the one hand, it allows for subsequent selection of matching keyframes from the latest operation stage and the historical operation stage, taking into account both the latest environmental features and historical reference features. This enables pose optimization using matching keyframes from multiple different operation cycles, improving the reliability of pose optimization. On the other hand, if the first and second operation cycles are two adjacent operation cycles, it allows for keyframe matching between the most recent operation cycle and the previous operation cycle, ensuring that the matched keyframes are as up-to-date as possible. This allows for pose optimization of the target sub-map using the matched keyframes, improving the accuracy of visual map updates.

[0074] (2) In some embodiments, the set of historical job keyframes and the set of current job keyframes are respectively the set of the first candidate keyframes of the first job cycle and the set of the third candidate keyframes of all historical job cycles. In this case, step 302 may specifically include the following steps 3021B~3023B: 3021B. Obtain the prior pose of the current processing keyframe.

[0075] 3021B. Based on the prior pose, determine key frames from the first set of key frames whose distance from the currently processed key frame is within a preset distance range, and use them as candidate key frames.

[0076] 3021B. The candidate keyframes are classified according to the operation cycle to obtain the first candidate keyframes of the first operation cycle and the third candidate keyframes of all historical operation cycles. The current operation keyframe set is the set of the first candidate keyframes, the historical operation keyframe set is the set of the third candidate keyframes, and the currently processed keyframe is the keyframe collected by the robot in the first operation cycle. The first operation cycle lags behind all historical operation cycles.

[0077] The third candidate keyframe refers to the candidate keyframes collected in all historical operation cycles.

[0078] The first job cycle is the job stage to which the current keyframe belongs, and the historical job cycle is the historical job stage that is earlier than the first job cycle. That is, the first job cycle lags behind all historical job cycles. Each candidate keyframe carries the corresponding job cycle identifier information, which can be directly used to complete the classification.

[0079] The implementation of steps 3021B to 3023B is similar to that of steps 3021A to 3023A. For details, please refer to the relevant explanations above. They will not be repeated here.

[0080] 303. Based on the current job keyframe set and the historical job keyframe set, select a preset number of keyframes and match them with the current processing keyframe to obtain the matching keyframe of the current processing keyframe.

[0081] Among them, the matching keyframe refers to the keyframe that matches the features of the currently processed keyframe.

[0082] The specific value of the preset quantity can be set according to the actual business scenario requirements. For example, a fixed value can be preset based on the computing power of the robot's embedded device and the requirements of the operation scenario. There is no restriction on the specific value of the preset quantity here.

[0083] Step 303 can limit the number of matching keyframes, reduce the computational load of subsequent pose optimization, reduce computational blocking problems caused by too many matching keyframes, and at the same time ensure the feature correlation between the selected matching keyframes and the currently processed keyframes, laying the foundation for the accuracy of pose optimization.

[0084] In some embodiments of step 303, the similarity between the first candidate keyframe and the currently processed keyframe can be calculated first, and the first candidate keyframes can be sorted from high to low according to feature similarity to obtain a first sorting sequence; the similarity between the second candidate keyframe and the currently processed keyframe can be calculated, and the second candidate keyframes can be sorted from high to low according to feature similarity to obtain a second sorting sequence; the keyframe with the highest similarity can be extracted alternately from the first sorting sequence and the second sorting sequence as the matching keyframe until the number of extracted matching keyframes is equal to the preset number. For example, the process involves first extracting the first matching keyframe from the first sorted sequence, then extracting the second matching keyframe from the second sorted sequence, and so on. During this process of alternately extracting the most similar keyframes from the first and second sorted sequences, if the number of extracted matching keyframes is less than a preset number and the keyframes in the first sorted sequence have been extracted, then the process continues to extract the most similar keyframes from the second sorted sequence as matching keyframes until the number of extracted matching keyframes equals the preset number. Alternatively, if the number of extracted matching keyframes is less than the preset number and the keyframes in the second sorted sequence have been extracted, then the process continues to extract the most similar keyframes from the first sorted sequence as matching keyframes until the number of extracted matching keyframes equals the preset number.

[0085] In some embodiments of step 303, on the one hand, it is considered that if too many keyframes are matched, subsequent pose optimization will take a lot of time, thus causing computational blockage; on the other hand, it is considered that the environment within the work area may change between two jobs. Therefore, according to a preset number, matching keyframes are selected from the keyframes collected in the first job cycle and the second job cycle, respectively. Figure 6 As shown, Figure 6 This is an illustrative diagram illustrating keyframe matching using keyframes from the first and second job cycles, respectively, provided in an embodiment of this application. In this case, step 303 may specifically include the following steps 3031A to 3033A: 3031A. Based on a preset quantity, the number of frames in the current job keyframe set, and the number of frames in the historical job keyframe set, determine the number of frames to be filtered in the first job cycle and the number of frames to be filtered in the second job cycle.

[0086] The preset quantity refers to the total number of matching keyframes set in advance. The number of frames selected in the first job cycle is the number of matching keyframes selected from the current job keyframe set, and the number of frames selected in the second job cycle is the number of matching keyframes selected from the historical job keyframe set. Ideally, the sum of the number of frames selected in the first job cycle and the number of frames selected in the second job cycle equals the preset quantity, and the number of frames selected in the first job cycle equals the number of frames selected in the second job cycle. Thus, in step 3031A, the allocation of the number of frames selected in the first and second job cycles based on the preset quantity follows the principle of making the quantity as equal as possible, while also taking into account the actual number of frames in each current job keyframe set and historical job keyframe set, thereby improving the rationality of the frame allocation.

[0087] For example, step 3031A may specifically include: detecting a first total number of frames in the current job keyframe set and a second total number of frames in the historical job keyframe set; if the first total number of frames is greater than or equal to half of the preset number and the second total number of frames is greater than or equal to half of the preset number, then half of the preset number is used as the number of frames to be filtered in the first job cycle and half of the preset number is used as the number of frames to be filtered in the second job cycle; or, if the first total number of frames is less than half of the preset number, then the first total number of frames is used as the number of frames to be filtered in the first job cycle and the difference between the preset number and the first total number of frames is used as the number of frames to be filtered in the second job cycle; or, if the second total number of frames is less than half of the preset number, then the second total number of frames is used as the number of frames to be filtered in the second job cycle and the difference between the preset number and the second total number of frames is used as the number of frames to be filtered in the first job cycle. In this way, by first detecting the first total number of frames in the current job key frame set and the second total number of frames in the historical job key frame set, and then comparing the actual number of frames in the two types of key frame sets (i.e., the current job key frame set and the historical job key frame set) with half of the preset number, the number of frames to be filtered is allocated according to the preset rules. This not only allows key frames of both types of job cycles to participate in matching, but also reduces the problem of single matching features caused by too many or too few frames in a single type.

[0088] The first total frame count refers to the total number of frames in the first candidate keyframe, and the second total frame count refers to the total number of frames in the second candidate keyframe.

[0089] For example, if the preset number is 6, and the first total number of frames in the current job key frame set is 5 and the second total number of frames in the historical job key frame set is 4, both of which are greater than or equal to half of the preset number (3 frames), then the number of frames to be filtered in the first job cycle is set to 3 and the number of frames to be filtered in the second job cycle is set to 3, for a total of 6 frames. If the first total number of frames in the current job key frame set is 2 frames and the second total number of frames in the historical job key frame set is 6 frames, and the first total number of frames is less than half of the preset number (3 frames), then the number of frames to be filtered in the first job cycle is set to 2 frames and the number of frames to be filtered in the second job cycle is set to 4 frames, for a total of 6 frames. If the first total number of frames in the current job key frame set is 6 frames and the second total number of frames in the historical job key frame set is 2 frames, and the second total number of frames is less than half of the preset number (3 frames), then the number of frames to be filtered in the second job cycle is set to 2 frames and the number of frames to be filtered in the first job cycle is set to 4 frames, for a total of 6 frames.

[0090] 3032A. Based on the number of frames filtered in the first job cycle, select the key frame with the highest feature similarity to the current processing key frame from the current job key frame set, and use it as the matching key frame.

[0091] For example, if the number of frames filtered in the first job cycle is equal to the first total number of frames, then all keyframes in the current job keyframe set are directly used as matching keyframes. If the number of frames filtered in the first job cycle is less than the first total number of frames, then keyframes with higher feature similarity to the currently processed keyframes are selected from the current job keyframe set according to the number of frames filtered in the first job cycle, and these are used as matching keyframes.

[0092] In some embodiments, global descriptors of the currently processed keyframe and each first candidate keyframe (i.e., each keyframe in the current job keyframe set) can be extracted first. The feature similarity between the two can be calculated using a feature matching algorithm. The first candidate keyframes are then sorted from high to low feature similarity. Based on the number of frames filtered in the first job cycle, the first candidate keyframe with the highest ranking is selected as the matching keyframe. The matching keyframes selected from the current job keyframe set in this way have a stronger feature correlation with the currently processed keyframe, can accurately reflect the latest job environment features, and provide feature references that fit the current scene for pose optimization.

[0093] In some embodiments, a global descriptor of a first candidate keyframe and a global descriptor of the currently processed keyframe can be obtained, wherein the first candidate keyframe is a keyframe in the current job keyframe set; a first search structure is constructed based on the global descriptor of the first candidate keyframe; the current job keyframe set is searched based on the global descriptor of the currently processed keyframe and the first search structure to obtain N1 keyframes with the highest feature similarity to the currently processed keyframe, which are used as the matching keyframes, wherein N1 is equal to the number of screening frames in the first job cycle. Therefore, firstly, it can improve efficiency. By constructing the first search structure and relying on its nearest neighbor retrieval characteristics, it can reduce the need to compare all first candidate keyframes one by one, thereby reducing redundant operations in feature similarity calculation to a certain extent, improving the overall efficiency of keyframe matching and screening, and adapting to the computing power requirements of robot embedded devices. Secondly, it can improve accuracy. By using global descriptors for feature representation, combined with the accurate nearest neighbor retrieval and frame number screening of the first search structure, the selected N1 matching keyframes have the highest feature similarity with the currently processed keyframe. This can accurately reflect the latest working environment features of the first work cycle (i.e., the work cycle to which the currently processed keyframe belongs), providing a reliable feature association reference that fits the current scene for subsequent pose optimization, further improving the accuracy and stability of pose optimization, and indirectly reducing problems such as robot positioning errors and missed tasks caused by pose optimization deviations.

[0094] Specifically, the first candidate keyframe is the candidate keyframe collected in the first work cycle.

[0095] The global descriptor refers to the global description of all feature points in a single keyframe. The specific type of the global descriptor is not limited in this embodiment of the application (e.g., the global descriptor can be extracted using the NetVLAD feature algorithm).

[0096] The first search structure refers to the search structure constructed based on the global descriptor of the first candidate keyframe. For example, a KDTree (K-Dimensional Tree) can be constructed based on the global descriptor of the first candidate keyframe as the first search structure. The first search structure can perform efficient nearest neighbor retrieval on high-dimensional data (i.e., the global descriptor of the first candidate keyframe), and can directly locate the keyframe with the highest similarity to the target data (i.e., the global descriptor of the currently processed keyframe). It does not require traversing all first candidate keyframes or calculating feature similarity one by one, which can greatly reduce retrieval time and improve the efficiency of filtering matching keyframes from the current set of keyframes.

[0097] For example, assuming the number of screening frames N1=3 in the first job cycle determined in step 3031A, and the current job keyframe set contains 5 first candidate keyframes (all collected in the first job cycle). First, using the same feature extraction algorithm (such as the NetVLAD feature extraction algorithm), the global descriptors of the 5 first candidate keyframes are obtained respectively, and the global descriptor of the currently processed keyframe is also extracted to ensure that the feature dimensions and description rules of the two are consistent; second, a KDTree is constructed based on the global descriptors of the 5 first candidate keyframes as the first search structure to organize the global descriptors of the 5 first candidate keyframes in an orderly manner; finally, the global descriptor of the currently processed keyframe is input into the first search structure, and relying on the nearest neighbor retrieval characteristics of the first search structure, the 3 first candidate keyframes with the highest feature similarity to the currently processed keyframe are directly retrieved and used as matching keyframes (N1=3, consistent with the number of screening frames in the first job cycle). In this way, without traversing all 5 first candidate keyframes or calculating feature similarity one by one, N1=3 matching keyframes can be retrieved efficiently, improving the feature correlation between the matching keyframes selected from the current job keyframe set and the currently processed keyframes.

[0098] 3033A. According to the number of frames selected in the second job cycle, select the key frame with the highest feature similarity to the current processing key frame from the set of historical job key frames, and use it as the matching key frame.

[0099] The specific implementation of step 3033A is similar to that of step 3032A. For example, if the number of frames filtered in the second job cycle is equal to the second total number of frames, then all keyframes in the historical job keyframe set are directly used as matching keyframes. If the number of frames filtered in the second job cycle is less than the second total number of frames, then according to the number of frames filtered in the second job cycle, keyframes with higher feature similarity to the currently processed keyframe are selected from the historical job keyframe set and used as matching keyframes.

[0100] In some embodiments, global descriptors of the current processing keyframe and each second candidate keyframe (i.e., each keyframe in the historical operation keyframe set) can be extracted first, and feature similarity can be calculated. The second candidate keyframes are then sorted from high to low feature similarity. Based on the number of frames selected in the second operation cycle, the top-ranked second candidate keyframe is selected as the second matching keyframe. Thus, step 3033A allows for the selection of matching keyframes from candidate keyframes in the historical operation cycle, overcoming the limitations of feature reference in a single operation cycle and adapting to the needs of dynamically changing operational scenarios. The second matching keyframe of the historical operation cycle (i.e., the second operation cycle) reflects the basic environmental characteristics of the operation area and complements the first matching keyframe. Even if the environment of the operation area undergoes local changes, it can provide a comprehensive feature association reference for pose optimization, improving the accuracy and stability of pose optimization.

[0101] In some embodiments, a global descriptor of a second candidate keyframe and a global descriptor of the currently processed keyframe are obtained, wherein the second candidate keyframe is a keyframe in the set of historical job keyframes; a second search structure is constructed based on the global descriptor of the second candidate keyframe; based on the global descriptor of the currently processed keyframe and the second search structure, the second candidate keyframe is retrieved to obtain N2 keyframes with the highest feature similarity to the currently processed keyframe, which are used as the matching keyframes, wherein N2 is equal to the number of screening frames in the second job cycle. Therefore, firstly, it can improve efficiency. By constructing a second search structure and relying on its nearest neighbor retrieval characteristics, it can reduce the need to compare all second candidate keyframes one by one, thereby reducing redundant operations in feature similarity calculation and improving the overall efficiency of second matching keyframe selection, which is suitable for the computing power requirements of robot embedded devices. Secondly, it can improve accuracy. By using global descriptors for feature representation and combining the accurate nearest neighbor retrieval and frame number selection of the second search structure, the selected N2 matching keyframes have the highest feature similarity with the currently processed keyframes. This can accurately reflect the basic environmental features of the second work cycle (i.e., the historical work cycle earlier than the current work cycle), providing a reliable feature association reference that fits the historical scene for subsequent pose optimization. This complements the matching keyframes selected from the current work keyframe set, further improving the accuracy and stability of pose optimization and indirectly reducing problems such as robot positioning errors and missed tasks caused by pose optimization deviations.

[0102] The second candidate keyframe is specifically the candidate keyframe collected in the second work cycle.

[0103] The second search structure refers to the search structure constructed based on the global descriptor of the second candidate keyframe. For example, a KDTree (K-Dimensional Tree) can be constructed based on the global descriptor of the second candidate keyframe as the second search structure. The second search structure can perform efficient nearest neighbor retrieval on high-dimensional data (i.e., the global descriptor of the second candidate keyframe), and can directly locate the keyframe with the highest similarity to the target data (i.e., the global descriptor of the currently processed keyframe). It does not require traversing all second candidate keyframes or calculating feature similarity one by one, which can significantly reduce retrieval time and improve the efficiency of filtering matching keyframes from the set of historical job keyframes.

[0104] For example, assuming the number of screening frames N2=3 for the second job cycle determined in step 3031A, and the historical job keyframe set contains 6 second candidate keyframes (all collected in the second job cycle). First, using the same feature extraction algorithm (such as the NetVLAD feature extraction algorithm), the global descriptors of the 6 second candidate keyframes are obtained respectively, and the global descriptor of the currently processed keyframe is also extracted to ensure that the feature dimensions and description rules of the two are consistent. Second, a KDTree is constructed based on the global descriptors of the 6 second candidate keyframes as a second search structure to organize the global descriptors of the 6 second candidate keyframes in an orderly manner. Finally, the global descriptor of the currently processed keyframe is input into the second search structure, and relying on the nearest neighbor retrieval characteristics of the second search structure, the 3 second candidate keyframes with the highest feature similarity to the currently processed keyframe are directly retrieved and used as matching keyframes (N2=3, consistent with the number of screening frames in the second job cycle). In this way, without traversing all 6 second candidate keyframes or calculating feature similarity one by one, N2=3 matching keyframes can be retrieved efficiently. This provides the feature correlation between the matching keyframes selected from the historical operation keyframe set and the currently processed keyframes, which can fully reflect the basic environmental features of the historical operation cycle (i.e., the second operation cycle) and complement the latest environmental features of the matching keyframes selected from the current operation keyframe set, thus adapting to the scenario requirements of dynamic changes in the operation area environment.

[0105] Thus, through steps 3031A to 3033A, the maximum number of keyframes that can be matched in the current processing keyframe can be limited to N (N being the preset number), thereby reducing the number of image pairs matched in the current processing keyframe, which in turn reduces the time required for subsequent pose optimization. This, in turn, reduces the obstruction of subsequent pose optimization to the calculation of other modules to a certain extent, reduces the loop lag problem, and further reduces the problem of positioning errors not being corrected in time due to loop lag. This, in turn, reduces the problem of missed robot operations caused by correction delay (such as reducing the problem of missed cutting by lawnmower robots).

[0106] To better understand how keyframe matching is performed in the embodiments of this application, a specific example is provided for illustration. Please refer to [link / reference]. Figure 7 , Figure 7 This is a flowchart illustrating an embodiment of keyframe matching. First, spatial filtering can be performed using prior poses in step 3022A (see reference). Figure 7 The text mentions "using prior pose for spatial filtering," and then steps 3023A to divide candidate keyframes within a distance range into candidate keyframes for the first job cycle and candidate keyframes for the second job cycle (see reference). Figure 7 The process involves "classifying candidate keyframes by job batch," thereby obtaining the current job keyframe set (i.e., the set of candidate keyframes for the first job cycle) and the historical job keyframe set (i.e., the set of candidate keyframes for the second job cycle). Then, through steps 3031A to 3033A, a first search structure and a second search structure are constructed for the current job keyframe set and the historical job keyframe set, respectively, to search for a preset number of matching keyframes (see reference). Figure 7 The phrase "Use global descriptors to search for top N keyframes" refers to the following process: During the search, N1 matching keyframes are selected from the current job's keyframe set based on the number of frames filtered in the first job cycle, and N2 matching keyframes are selected from the historical job's keyframe set based on the number of frames filtered in the second job cycle (see reference). Figure 7 (This refers to "selecting candidate keyframes while considering different job batches"). Therefore, by performing keyframe matching through steps 3021A~3023A and steps 3031A~3033A, the total number of matched keyframes can be limited, and subsequent computational blocking can be reduced. At the same time, the latest features and historical reference features of the job environment can be taken into account, making the feature coverage of the matched keyframes more comprehensive. This effectively improves the rationality and adaptability of keyframe matching and enhances the accuracy of subsequent pose optimization.

[0107] 304. Based on the matching relationship between the matching keyframe and the currently processed keyframe, the pose of the target sub-map is optimized, and the currently processed keyframe is added to the first keyframe set to obtain the second keyframe set of the target sub-map.

[0108] In some embodiments, pose optimization, pose graph optimization, and other methods can be used to optimize the pose of the target sub-map based on the matching relationship between the matching keyframe and the currently processed keyframe. BA optimization, or Bundle Adjustment, is an algorithm that optimizes camera pose and spatial point coordinates by minimizing reprojection error. BA optimization is a global fine-tuning method that achieves accurate global pose correction of the sub-map by jointly minimizing the reprojection error between the keyframe pose and the 3D spatial points (i.e., the 3D spatial points corresponding to the feature points matched between the matching keyframe and the currently processed keyframe), thereby reducing cumulative drift. In this embodiment, the reprojection error refers to the deviation between the pixel coordinates of the feature points in the currently processed keyframe projected onto the imaging plane of the matching keyframe by the camera pose and the corresponding feature points in the matching keyframe. The magnitude of the reprojection error directly reflects the accuracy of the camera pose. Among them, pose graph optimization is a lightweight global optimization method. Using keyframes as nodes and inter-frame pose transformation constraints as edges, it primarily optimizes the relative pose relationships between keyframes without additionally adjusting the coordinates of points in 3D space. Its computational load is far less than that of basic pose optimization (BA), resulting in low computational cost, fast iteration speed, and a balance between optimization efficiency and sub-map pose accuracy, adapting to real-time map updates on the robot's end-user device. By optimizing the pose of the target sub-map based on the matching relationship between the matched keyframes and the currently processed keyframes, it can correct deviations in the prior pose caused by robot movement and sensor errors during acquisition, improving the accuracy of the poses of each keyframe in the target sub-map.

[0109] In some embodiments, after the pose optimization of the target sub-map is completed based on the matching relationship between the matching keyframe and the currently processed keyframe, the currently processed keyframe carrying the optimized pose, feature point information, and acquisition timestamp is added to the first keyframe set already stored in the target sub-map to form an updated second keyframe set. The second keyframe set is the latest keyframe storage carrier of the target sub-map, and all the keyframes it contains can be valid keyframes after pose optimization, which can provide accurate pose and feature references for subsequent visual map updates and robot localization.

[0110] As can be seen from the above, firstly, by optimizing the pose of the target sub-map based on the matching relationship between the matched keyframes and the currently processed keyframes, and adding the currently processed keyframes to the first set of keyframes already stored in the target sub-map to obtain the second set of keyframes, the environmental information of the latest job-collected keyframes can be used to update the visual global map, ensuring that the visual global map can be updated with environmental changes and ensuring positioning accuracy. Secondly, by selecting a preset number of keyframes based on the current job keyframe set and the historical job keyframe set to match with the currently processed keyframes, the number of matched keyframes can be limited, reducing the computational load of subsequent pose optimization and reducing computational blocking problems caused by too many matched keyframes. This reduces the time consumption of visual map update processing, improves the update speed of visual map, and improves the robot's positioning accuracy. Thirdly, by selecting a preset number of keyframes based on the current job keyframe set and the historical job keyframe set to match with the currently processed keyframes, pose optimization can be performed using matched keyframes from multiple different job cycles. This allows for pose optimization by combining the latest and historical environments, improving the reliability of pose optimization, thereby improving the accuracy of the target sub-map, and thus, to a certain extent, improving the robot's positioning accuracy. Therefore, this application can improve the robot's positioning accuracy while ensuring that the visual global map can be updated with changes in the environment.

[0111] Furthermore, the visual map update processing method may also include step 305 (not shown in the figure): 305. If the second set of keyframes meets the preset grid update conditions, then the keyframes in each grid area of ​​the target sub-map are simplified.

[0112] The target sub-map corresponds to multiple grid regions. The preset grid update conditions include at least one of the following: the number of frames in the second keyframe set is greater than a preset frame number threshold, or the number of feature points in the second keyframe set is greater than a preset feature point number threshold. By triggering a simplification process on the keyframes in each grid region of the target sub-map when either the number of frames in the second keyframe set is greater than the preset frame number threshold or the number of feature points in the second keyframe set is greater than the preset feature point number threshold, the number of feature points and keyframes in the target sub-map is kept within a certain data range and fluctuates dynamically.

[0113] The specific values ​​of the preset frame count threshold and the preset feature point count threshold can be set according to actual business needs. There are no restrictions on the specific values ​​of the preset frame count threshold and the preset feature point count threshold here.

[0114] Please refer to Figure 8 , Figure 8This is an illustrative diagram illustrating the updating of the visual map using keyframes collected in the first work cycle in an embodiment of this application. Through steps 301 to 304, the currently processed keyframes (i.e., keyframes collected in the first work cycle) are continuously added to the sub-map of the visual global map. In step 305, the keyframes in each grid area of ​​the sub-map are simplified, thereby replacing the keyframes collected in the historical work cycles with the keyframes collected in the first work cycle. This ensures that the visual global map maintains the latest environmental feature information, improving the accuracy of positioning using the visual map and the success rate of repositioning.

[0115] There are several ways to implement step 305, including, for example: (1) In some embodiments, each grid region retains the latest keyframe. For example, step 305 may specifically include the following steps 3051A to 3054A: 3051A. If the second set of keyframes meets the preset grid update conditions, the target sub-map is divided into multiple grid regions in the preset coordinate plane according to the preset size.

[0116] Here, the preset coordinate plane refers to a coordinate plane constructed based on the coordinate axes of a three-dimensional spatial coordinate system. For example, suppose the robot is in the X-axis of the three-dimensional spatial coordinate system. w Y w Work on a plane, i.e., the work area corresponds to X. w Y w A plane, then X can be... w Y w The plane is used as the preset coordinate plane.

[0117] The preset size is the pre-defined planar size of a single grid area on a preset coordinate plane, used to limit the spatial range of a single grid. The specific value of the preset size can be set according to actual business needs, and there is no restriction on the specific value of the preset size here.

[0118] Among them, the grid area is an independent planar area formed after the target sub-map is split. Each grid area corresponds to a fixed spatial range on the preset coordinate plane. All grid areas are connected to each other, do not overlap, and cover the entire target sub-map.

[0119] For example, it is determined whether the second keyframe set of the target sub-map meets the preset grid update conditions, that is, whether the number of frames in the second keyframe set is greater than the preset frame number threshold, or whether the number of feature points it contains is greater than the preset feature point number threshold. If either of these conditions is met, it is considered to meet the preset grid update conditions. If the preset grid update conditions are met, the target sub-map is evenly divided into multiple interconnected and non-overlapping grid regions on the preset coordinate plane according to the preset size. All grid regions together cover the entire planar range of the target sub-map, completing the grid division for subsequent keyframe attribution determination.

[0120] For example, suppose the target sub-map corresponds to the lawn operation area of ​​a lawnmower facing a certain direction. The preset coordinate plane is selected as the XwYw plane, the preset size is set to 1 meter × 1 meter, the preset frame threshold is set to 30 frames, and the preset feature point threshold is set to 1000. After judgment, the second keyframe set of the current target sub-map contains 50 keyframes (more than 30 frames), which meets the preset grid update conditions. At this time, the target sub-map is evenly divided on the XwYw plane according to the preset size of 1 meter × 1 meter, forming multiple interconnected and non-overlapping grid areas. Each grid area corresponds to a fixed range of 1 meter on both the Xw and Yw axes. For example, the spatial range of some grid areas are Xw∈[0,1], Yw∈[0,1], Xw∈[1,2], Yw∈[0,1], Xw∈[0,1], Yw∈[1,2], etc. All grid areas together cover the entire target sub-map, completing the grid division of the target sub-map.

[0121] 3052A. Based on the storage coordinates of each keyframe in the second keyframe set and the spatial range of each grid region, determine the grid region to which each keyframe belongs in the second keyframe set.

[0122] The stored coordinate position refers to the coordinate position corresponding to each keyframe in the second keyframe set. Specifically, it is the position (Xw, Yw, Zw) of the camera coordinate system relative to the three-dimensional spatial coordinate system when the keyframe is acquired. It is stored in the target sub-map along with the keyframe. In practice, the pose of the keyframe can be stored in the target sub-map along with the keyframe. The stored coordinate pose of the keyframe can be determined by reading the pose of the keyframe.

[0123] The spatial location range of the grid area is the coordinate range of each grid area on the preset coordinate plane (XwYw plane), which is determined by the preset size and division rules in step 3051A.

[0124] Among them, the assigned grid region refers to the grid region in which the storage coordinate position of each keyframe in the second keyframe set falls, and each keyframe corresponds to only one assigned grid region.

[0125] For example, firstly, the stored coordinate positions (Xw, Yw, Zw) of each key frame j in the second key frame set are extracted one by one from the stored data of the target sub-map; secondly, the spatial position range of each grid region divided in step 3051A on the preset coordinate plane (XwYw plane) is extracted; finally, for key frame j, the Xw and Yw components of its stored coordinate position are extracted, and the Xw and Yw components are compared one by one with the spatial position range of all grid regions to determine the grid region in which the Xw and Yw components fall. This grid region is the grid region to which key frame j belongs, thus completing the determination of the belonging of all key frames.

[0126] To facilitate understanding, let's continue with the example from step 3051A. For instance, the target sub-map has been divided into multiple 1m x 1m grid regions, where the spatial ranges of three grid regions are: grid 1 (Xw ∈ [2,3], Yw ∈ [4,5]), grid 2 (Xw ∈ [3,4], Yw ∈ [4,5]), and grid 3 (Xw ∈ [2,3], Yw ∈ [5,6]). Assuming the storage coordinates of keyframe j in the second keyframe set are (3.2m, 4.6m, 0.2m), then the Xw (3.2m) and Yw (4.6m) of keyframe 2 fall within the range of grid 2, thus determining grid 2 as the grid region to which keyframe j belongs, and so on.

[0127] 3053A. Based on the grid region to which each key frame belongs in the second key frame set, select the key frame with the latest acquisition timestamp in each grid region as the retained key frame.

[0128] The acquisition timestamp refers to the time information when the robot acquires each keyframe. It is stored in the target sub-map along with the pose, feature points, and other data of the keyframes and is used to represent the acquisition order of the keyframes.

[0129] Among them, retained keyframes refer to the keyframes that are filtered and retained in each grid area, serving as reference frames for the environmental features of that grid area.

[0130] For example, firstly, all keyframes in the second keyframe set can be grouped according to the grid region to which each keyframe belongs. Keyframes belonging to the same grid region are grouped into one keyframe group, and each grid region corresponds to a unique keyframe group. Then, for each grid region's keyframe group, the acquisition timestamps of all keyframes in the group are extracted one by one. The acquisition timestamps of each keyframe in the group are compared, and the keyframe with the latest acquisition timestamp is selected and determined as the retained keyframe for that grid region. For example, suppose the keyframe group of grid 1 contains keyframe 1 (acquisition timestamp: 2026-02-10, 09:05:00), keyframe 4 (2026-02-10, 14:20:00), and keyframe 5 (2026-02-10, 10:10:00), and the group of grid 2 contains keyframe 2 (2026-02-10, 10:30:00) and keyframe 6 (2026-02-10, 09:40:00). Comparing the timestamps within the keyframe group of grid 1, keyframe 4 in grid 1 has the latest timestamp and is determined to be the retained keyframe of grid 1; comparing the timestamps within the keyframe group of grid 2, keyframe 2 in grid 2 has the latest timestamp and is determined to be the retained keyframe of grid 2; and so on.

[0131] 3054A. Delete all keyframes in each of the grid regions except for the retained keyframes, so as to simplify the keyframes in each grid region of the target sub-map.

[0132] Among them, other keyframes refer to all keyframes within the keyframe group of each grid region, excluding the retained keyframes. These keyframes were acquired earlier than the retained keyframes and are considered redundant keyframes within the grid region. Thus, by simplifying the process and deleting redundant keyframes within the grid region, the total number of keyframes in the target sub-map can be reduced, thereby reducing memory and disk usage.

[0133] For example, for each grid region of the target sub-map, after determining the keyframes to be retained for that region, all other keyframes except the retained keyframes are directly deleted from the keyframe group of that grid region. After all grid regions have completed the redundant keyframe deletion operation, the keyframes in each grid region of the target sub-map are simplified, retaining only the latest environmental feature reference frames of each grid region. This allows for direct determination of the age of keyframes in each grid region of the target sub-map for visual map updates. For example, the second keyframe set of the target sub-map before simplification contains 50 keyframes. Assuming the target sub-map is divided into 12 grid regions: grid 1, grid 2, ..., grid 12, where the retained keyframe of grid 1 is keyframe 4, and the other keyframes are keyframe 1 and keyframe 5, then keyframes 1 and 5 in grid 1 are deleted; the retained keyframe of grid 2 is keyframe 2, and the other keyframes are keyframe 6, then keyframe 6 in grid 2 is deleted, and the remaining grid regions are deleted according to the same rules. Before simplification, the second keyframe set of the target submap contained 50 keyframes. After simplification, only one keyframe of each grid region is retained, reducing the total number of frames to 12. This significantly reduces redundant data, thereby allowing the memory required by the target submap to be dynamically controlled within a certain range, reducing the storage footprint of the robot's embedded devices.

[0134] (2) In some embodiments, each grid region retains the latest M keyframes. For example, step 305 may specifically include the following steps 3051B~3054B: 3051B. If the second set of keyframes meets the preset grid update conditions, the target sub-map is divided into multiple grid regions in the preset coordinate plane according to the preset size.

[0135] 3052B. Based on the storage coordinates of each keyframe in the second keyframe set and the spatial range of each grid region, determine the grid region to which each keyframe belongs in the second keyframe set.

[0136] 3053B. Based on the grid region to which each key frame belongs in the second key frame set, select the M key frames with the latest acquisition timestamp in each grid region as retained key frames.

[0137] Where M is a positive integer pre-set based on actual business needs. The specific value can be adjusted according to the storage computing power of the robot's embedded device and the complexity of the working area environment. For example, M can be set to values ​​such as 2, 3, and 5 to balance the richness of environmental feature references and the rationality of storage resources.

[0138] 3054B. Delete all keyframes in each of the grid regions except for the retained keyframes, so as to simplify the keyframes in each grid region of the target sub-map.

[0139] The implementation of steps 3051B to 3054B is similar to that of steps 3051A to 3054A. For details, please refer to the relevant explanations above. They will not be repeated here.

[0140] In this way, by retaining the latest M keyframes in each grid area, compared with the single-frame retention method, richer environmental feature references can be provided for subsequent visual map updates and pose optimization, adapting to complex environments and uneven distribution of feature points, and improving the accuracy and stability of pose optimization. At the same time, by limiting the number of retained frames to M, the overall data volume of the target sub-map can be effectively controlled, and the rational use of storage resources can reduce the storage pressure caused by too many keyframes.

[0141] Therefore, by adding the currently processed keyframe to the first keyframe set already stored in the target sub-map, a second keyframe set is obtained. This allows the environmental information from the latest acquired keyframes to be used to update the visual global map, ensuring that the visual global map can update with environmental changes and maintaining positioning accuracy. When the second keyframe set meets preset grid update conditions—that is, when the number of frames in the second keyframe set exceeds a preset frame count threshold, or when the number of feature points in the second keyframe set exceeds a preset feature point count threshold—simplification processing of keyframes within each grid region of the target sub-map is triggered. This keeps the number of feature points and keyframes in the target sub-map dynamically fluctuating within a certain data range, thus reducing the continuous increase in memory and disk usage and minimizing memory or disk storage overload. Therefore, this application can reduce the continuous increase in memory and disk usage and minimize memory or disk storage overload while ensuring that the visual global map can update with environmental changes.

[0142] Those skilled in the art will understand that all or part of the steps in the above-described visual map update processing method can be accomplished by instructions, or by controlling related hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0143] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the visual map update processing methods provided in embodiments of this application.

[0144] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] In the above embodiments of the mobile robot and computer-readable storage medium, the descriptions of each embodiment have different focuses. For parts not described in detail in a particular embodiment, please refer to the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the computer-readable storage medium, mobile robot, and their corresponding units described above can be referred to the description of the visual map update processing method in the above embodiments, and will not be repeated here.

[0146] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0147] The foregoing has provided a detailed description of a visual map update processing method, a mobile robot, and a computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A visual map update processing method, characterized in that, The method includes: From the sub-maps of the visual global map, determine the target sub-map associated with the robot's current processing keyframe; Based on the first set of keyframes already stored in the target sub-map, and combined with the prior pose of the currently processed keyframe, candidate keyframes are filtered within a preset range. The candidate keyframes include the current job keyframe set and the historical job keyframe set. Based on the current job keyframe set and the historical job keyframe set, a preset number of keyframes are selected and matched with the current processing keyframe to obtain the matching keyframe of the current processing keyframe. Based on the matching relationship between the matching keyframe and the currently processed keyframe, the pose of the target sub-map is optimized, and the currently processed keyframe is added to the first keyframe set to obtain the second keyframe set of the target sub-map.

2. The visual map update processing method according to claim 1, characterized in that, The visual global map is divided into multiple sub-maps according to the working orientation, and each sub-map corresponds to an orientation angle range; Determining the target sub-map associated with the robot's current processing keyframe from among the sub-maps of the visual global map includes: Obtain the prior pose of the current keyframe being processed; Based on the orientation angle range of each sub-map and the prior pose, the target sub-map associated with the current processing keyframe is determined.

3. The visual map update processing method according to claim 1, characterized in that, The process of filtering candidate keyframes within a preset range based on the first set of keyframes already stored in the target sub-map, and in conjunction with the prior pose of the currently processed keyframe, includes: Obtain the prior pose of the current keyframe being processed; Based on the prior pose, key frames that are within a preset distance range from the current key frame are determined from the first key frame set and used as candidate key frames. The candidate keyframes are classified according to the operation cycle to obtain candidate keyframes for the first operation cycle and candidate keyframes for the second operation cycle. The current operation keyframe set is the set of candidate keyframes for the first operation cycle, the historical operation keyframe set is the set of candidate keyframes for the second operation cycle, and the currently processed keyframe is the keyframe collected by the robot in the first operation cycle, which lags behind the second operation cycle.

4. The visual map update processing method according to claim 3, characterized in that, The step of selecting a preset number of keyframes based on the current job keyframe set and the historical job keyframe set and matching them with the current processing keyframe to obtain the matching keyframes for the current processing keyframe includes: Based on a preset number, the number of frames in the current job keyframe set, and the number of frames in the historical job keyframe set, the number of frames to be filtered in the first job cycle and the number of frames to be filtered in the second job cycle are determined. Based on the number of frames filtered in the first job cycle, select the key frame with the highest feature similarity to the current processing key frame from the current job key frame set, and use it as the matching key frame; Based on the number of frames selected in the second job cycle, key frames with higher feature similarity to the currently processed key frame are selected from the set of historical job key frames and used as the matching key frames.

5. The visual map update processing method according to claim 4, characterized in that, The step of selecting keyframes with higher feature similarity to the currently processed keyframes from the current task keyframe set according to the number of frames selected in the first task cycle, and using these as the matching keyframes, includes: Obtain the global descriptor of the first candidate keyframe and the global descriptor of the currently processed keyframe, wherein the first candidate keyframe is a keyframe in the current job keyframe set; Based on the global descriptor of the first candidate keyframe, a first search structure is constructed; Based on the global descriptor of the current processing key frame and the first search structure, the current job key frame set is retrieved to obtain the N1 key frames with the highest feature similarity to the current processing key frame, which are used as the matching key frames, where N1 is equal to the number of screening frames in the first job cycle.

6. The visual map update processing method according to claim 4, characterized in that, The step of selecting keyframes from the historical keyframe set that have the highest feature similarity to the currently processed keyframe according to the number of frames selected in the second job cycle, and using them as the matching keyframes, includes: Obtain the global descriptor of the second candidate keyframe and the global descriptor of the currently processed keyframe, wherein the second candidate keyframe is a keyframe in the set of historical job keyframes; Based on the global descriptor of the second candidate keyframe, a second search structure is constructed; Based on the global descriptor of the current processing keyframe and the second search structure, the second candidate keyframes are retrieved to obtain the N2 keyframes with the highest feature similarity to the current processing keyframe, which are then used as the matching keyframes, where N2 is equal to the number of screening frames in the second job cycle.

7. The visual map update processing method according to claim 4, characterized in that, The step of determining the number of frames to be filtered in the first job cycle and the number of frames to be filtered in the second job cycle based on a preset number, the number of frames in the current job keyframe set, and the number of frames in the historical job keyframe set includes: Detect the first total number of frames in the current job keyframe set and the second total number of frames in the historical job keyframe set; If the first total number of frames is greater than or equal to half of the preset number, and the second total number of frames is greater than or equal to half of the preset number, then half of the preset number is used as the number of frames to be filtered in the first job cycle, and half of the preset number is used as the number of frames to be filtered in the second job cycle. Alternatively, if the first total number of frames is less than half of the preset number, then the first total number of frames is used as the number of frames to be filtered in the first job cycle, and the difference between the preset number and the first total number of frames is used as the number of frames to be filtered in the second job cycle. Alternatively, if the second total number of frames is less than half of the preset number, then the second total number of frames is used as the number of frames to be filtered in the second job cycle, and the difference between the preset number and the second total number of frames is used as the number of frames to be filtered in the first job cycle.

8. The visual map update processing method according to claim 1, characterized in that, The method further includes: If the second keyframe set meets the preset grid update conditions, then the keyframes in each grid region of the target sub-map are simplified. The target sub-map corresponds to multiple grid regions. The preset grid update conditions include at least one of the following: the number of frames in the second keyframe set is greater than a preset frame number threshold, or the number of feature points in the second keyframe set is greater than a preset feature point number threshold.

9. The visual map update processing method according to claim 2, characterized in that, The simplification process for keyframes within each grid region of the target sub-map includes: The target sub-map is divided into multiple grid areas on a preset coordinate plane according to a preset size; Based on the storage coordinates of each key frame in the second key frame set and the spatial range of each grid region, the grid region to which each key frame belongs in the second key frame set is determined. Based on the grid region to which each key frame belongs in the second key frame set, the key frame with the latest acquisition timestamp is selected as the retained key frame within each grid region. Delete all keyframes except the retained keyframes within each grid area to simplify the keyframe processing in each grid area of ​​the target sub-map.

10. A mobile robot, characterized in that, The system includes a camera, a processor, and a memory, wherein the memory stores a computer program, and the processor executes the visual map update processing method as described in any one of claims 1 to 9 when it invokes the computer program in the memory.

11. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to execute the visual map update processing method according to any one of claims 1 to 9.

Images