Visual positioning method, self-moving device and storage medium

Abstract

The invention relates to the technical field of robot navigation, and discloses a visual positioning method, self-moving equipment and a storage medium, and the method comprises the steps: obtaining a plurality of frames of environment images containing a time sequence relation; analyzing the multi-frame environment image to obtain an anomaly recognition result in the environment image; extracting effective features based on the multi-frame environment image to obtain a first effective feature set; screening the first effective feature set according to an anomaly recognition result, and removing false features corresponding to the image anomaly to obtain a second effective feature set after the anomaly is removed; and calculating pose information of the self-moving device based on the second effective feature set. According to the technical scheme, the accuracy and robustness of visual positioning are improved, and navigation errors caused by image abnormity are reduced.

Description

Technical Field

[0001] This application relates to the field of robot navigation technology, specifically to a visual positioning method, a self-moving device, and a storage medium. Background Technology

[0002] In the field of outdoor service robots, especially in the operating environment of self-moving devices such as lawnmowers, the dynamic characteristics of outdoor scenes make the quality of camera input images easily affected, seriously impacting the system's mapping accuracy and operational stability.

[0003] Specifically, during operation, the lens surface of the equipment is frequently and momentarily or continuously obstructed by objects such as branches, leaves, water droplets, mud, grass clippings, or insects. These obstructions themselves exhibit rich texture features, making it difficult for traditional feature extraction algorithms to distinguish them from real-world environmental features, resulting in a sharp decrease in the number of effective feature points in the image. When the feature point density falls below the system's operating threshold, visual tracking is interrupted, and the system is forced to switch to relocalization mode. Summary of the Invention

[0004] This application provides a visual positioning method, a self-moving device, and a storage medium, which improves the accuracy and robustness of visual positioning and reduces navigation errors caused by image anomalies.

[0005] In a first aspect, this application provides a visual positioning method applied to a self-moving device. The method includes: acquiring multiple frames of environmental images containing time-series relationships; analyzing the multiple frames of environmental images to obtain anomaly recognition results in the environmental images; extracting effective features based on the multiple frames of environmental images to obtain a first effective feature set; filtering the first effective feature set according to the anomaly recognition results, removing false features corresponding to image anomalies, to obtain a second effective feature set after anomaly removal; and calculating the pose information of the self-moving device based on the second effective feature set.

[0006] Secondly, this application provides a self-moving device, including: an image acquisition component, a movement component, a power component, a memory, and a processor. The image acquisition component is used to acquire environmental images, the movement component is used to record the pose of the self-moving device and perform self-movement, the power component is used to provide power to the self-moving device, the memory and the processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual positioning method of the first aspect or any corresponding embodiment described above.

[0007] Thirdly, this application provides a visual positioning device, comprising: an acquisition module for acquiring multiple frames of environmental images containing time-series relationships; a recognition module for analyzing the multiple frames of environmental images to obtain anomaly recognition results in the environmental images; an extraction module for extracting effective features based on the multiple frames of environmental images to obtain a first effective feature set; a rejection module for filtering the first effective feature set according to the anomaly recognition results to reject false features corresponding to image anomalies, thereby obtaining a second effective feature set after anomaly rejection; and a calculation module for calculating the pose information of a self-moving device based on the second effective feature set.

[0008] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the visual positioning method of the first aspect or any corresponding embodiment described above.

[0009] Fifthly, this application provides a computer program product, including computer instructions for causing a computer to execute the visual positioning method of the first aspect or any corresponding embodiment described above.

[0010] As can be seen from the above, the visual positioning method and self-moving device provided in this application acquire multiple frames of environmental images, analyze the anomaly recognition results, extract and filter effective features, and eliminate false features, thereby accurately calculating pose information, improving the accuracy and robustness of visual positioning, and reducing navigation errors caused by image anomalies. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a first type of visual positioning method according to an embodiment of this application; Figure 2 This is a schematic diagram of a second type of visual positioning method according to an embodiment of this application; Figure 3 This is a schematic diagram of the third process of the visual positioning method according to an embodiment of this application; Figure 4 This is a structural block diagram of a visual positioning device according to an embodiment of this application; Figure 5 This is a schematic diagram of a self-moving device according to an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0015] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0016] Traditional visual SLAM technologies face challenges in mapping accuracy and system robustness when applied to self-moving devices such as outdoor lawnmowers. The complex and variable outdoor environment leads to a significant deterioration in the quality of camera input images. Specifically, momentary lens occlusions (such as branches, leaves, and water droplets) are easily misidentified as stable environmental features, creating false feature points, introducing pose estimation errors, and reducing mapping accuracy. Simultaneously, drastic changes in lighting (such as glare and overexposure) cause a sharp reduction in the number of effective environmental feature points, making it easy for the SLAM system to lose track of them, thus affecting operational efficiency.

[0017] To address this issue, this application proposes a visual positioning method applied to self-moving devices. The visual positioning method refers to the technical process of determining the position and orientation of a device in three-dimensional space by analyzing image data. Self-moving devices refer to devices capable of autonomous movement and performing specific tasks, such as outdoor lawnmower robots and unmanned delivery vehicles. Figure 1 As shown, the visual positioning method includes: Step S101: Obtain multi-frame environmental images containing time-series relationships.

[0018] Among them, multi-frame environmental images with time-series relationships refer to a series of images continuously acquired by the image acquisition component at different time points. These images have a temporal sequence, reflecting the changes in the environment over time.

[0019] Specifically, acquiring multi-frame environmental images containing a time-series relationship can be achieved through an image acquisition component configured on a self-moving device. For example, the image acquisition component can continuously capture environmental images at a preset frame rate and store these images in chronological order of capture time. These images can be color images or grayscale images, and their continuity helps to capture dynamic changes in the environment.

[0020] Step S102: Analyze the multiple frames of environmental images to obtain the anomaly recognition results in the environmental images.

[0021] Among them, the anomaly identification result refers to the part of the image that does not conform to the characteristics of the normal environment by analyzing the environmental image, such as the image area caused by factors such as occlusion, strong light, overexposure, etc.

[0022] Specifically, multiple frames of environmental images are analyzed to obtain anomaly identification results. This analysis process can be carried out in various ways. For example, a method based on image pixel statistics can be used to identify areas with excessively high or low brightness as areas of abnormal lighting by calculating the average brightness or contrast of local areas of the image; or, by detecting suddenly appearing, irregular areas in the image that do not match the background texture, these areas can be identified as occluded areas.

[0023] Step S103: Extract effective features based on multi-frame environmental images to obtain the first effective feature set.

[0024] The first effective feature set refers to the set of visual feature points initially extracted from multiple frames of environmental images. These feature points may contain false features caused by image anomalies.

[0025] Specifically, this feature extraction process can employ various known feature extraction algorithms. For example, the Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature Transform (SURF), or Oriented Fast and Rotation Invariant Feature Transform (ORB) algorithms can be applied to each frame of the environmental image to detect and describe key points in the image. These key points and their descriptors are collected to form a first effective feature set.

[0026] Step S104: Based on the anomaly identification results, the first effective feature set is filtered to remove false features corresponding to image anomalies, thereby obtaining the second effective feature set after removing anomalies.

[0027] False features refer to visual feature points in an image that are not generated by real environmental objects, but rather by factors such as lens obstructions or abnormal lighting, and are easily misidentified as stable environmental features. The second effective feature set refers to the set of visual feature points that, based on the first effective feature set, has been filtered out after anomaly identification results, eliminating false features. The feature points in this set have higher reliability.

[0028] Specifically, this screening process aims to remove unreliable feature points generated by anomalous regions. For example, a rule can be established that if the location coordinates of a feature point fall within a previously identified anomalous region (such as an occluded region or an area with abnormal lighting), then the feature point is determined to be a false feature and removed from the first set of valid features. In this way, the feature set can be initially purified.

[0029] Step S105: Based on the second effective feature set, calculate the pose information of the self-moving device. Here, pose information refers to the combined data of the self-moving device's position and orientation in three-dimensional space.

[0030] Specifically, this pose calculation process can utilize various geometric methods. For example, a visual odometry method based on feature point matching can be used, matching the second set of valid features in the current frame with the feature set in the previous frame, and estimating the relative motion of the self-moving device between the two frames through the geometric relationship between the matched points. Alternatively, if a pre-constructed environment map exists, the second set of valid features in the current frame can be matched with 3D points in the map, and the global pose of the self-moving device can be directly calculated using the Perspective-n-Point (PnP) algorithm.

[0031] It is understood that this application performs anomaly identification on multiple frames of environmental images acquired by the self-mounted device, and filters the extracted feature points based on the identification results, effectively eliminating false feature points caused by lens occlusion and changes in lighting. This ensures the purity and reliability of the feature set used for pose calculation, significantly improving the accuracy and robustness of visual positioning of the self-mounted device in complex outdoor environments, and reducing operation interruptions and the need for manual intervention due to positioning failures.

[0032] In some embodiments, this application further proposes a visual positioning method, the method comprising: Step S201: Obtain multi-frame environmental images containing time-series relationships. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0033] Step S202: Analyze the multiple frames of environmental images to obtain the anomaly recognition results in the environmental images.

[0034] Specifically, it includes: Step S2021: Preprocess the multiple frames of environmental images to obtain corresponding color image sequences and grayscale image sequences.

[0035] Specifically, preprocessing can include operations such as denoising, brightness / contrast adjustment, and size normalization of the original image to eliminate noise and inconsistencies that may have been introduced during acquisition. For example, Gaussian filters or median filters can be applied to smooth the image, reducing the interference of random noise on subsequent feature extraction. Simultaneously, converting the original color image to grayscale can highlight the texture and structural information of the image, while the color image retains rich color and lighting information. Color image sequences are better able to capture lighting-related anomalies (such as glare and overexposure), while grayscale image sequences are more focused on identifying the texture features of occluders (such as stains or water droplets on the lens). Ensuring that the generated color image sequence and grayscale image sequence maintain a strict correspondence in timestamps and frame numbers is fundamental for the correct processing by the subsequent neural network model.

[0036] Step S2022: Input the color image sequence and grayscale image sequence into the lightweight multi-task neural network model.

[0037] Specifically, preprocessed color and grayscale image sequences are used as input to the model, enabling it to utilize information from two different modalities simultaneously for anomaly detection. Lightweight design refers to a model that maintains high recognition performance while having a small size and low computational complexity. For example, lightweight backbone networks such as MobileNet, ShuffleNet, and EfficientNet can be used as the encoder part of the model, or the model can be optimized through techniques such as model pruning and quantization to meet the real-time processing needs of mobile devices in environments with limited computing resources. Multi-task neural network models mean that the model can handle multiple related recognition tasks simultaneously, such as simultaneously segmenting occluded regions and regions with abnormal lighting, avoiding the inefficiency of needing to run multiple models independently or process them step-by-step.

[0038] Step S2023: The lightweight multi-task neural network model outputs the anomaly recognition results of the environmental image. The anomaly recognition results include at least occluded areas and / or areas with abnormal lighting.

[0039] Specifically, the model outputs the identified anomaly information in a structured form through its decoder. For example, the output could be a pixel-level binary mask, where each pixel is labeled as belonging to a normal or anomaly region (such as an occluded area or an area with abnormal lighting), or it could output the bounding box coordinates of the anomaly region. An occluded area can specifically refer to an image area covered by objects such as leaves, water droplets, mud, grass clippings, insects, or fingerprints attached to the lens. An abnormal lighting region can refer to areas in the image caused by drastic changes in lighting, such as glare, bright spots, halos, or large areas of overexposure or underexposure. By clearly distinguishing the anomaly type, subsequent visual features can be processed in a targeted manner, effectively eliminating false features and improving the accuracy and robustness of localization.

[0040] It is understood that the technical solution in this embodiment preprocesses multiple frames of environmental images to obtain corresponding color image sequences and grayscale image sequences. This fully utilizes the illumination information of the color images and the texture information of the grayscale images, providing comprehensive and more robust input data for anomaly recognition and effectively addressing image quality issues in complex outdoor environments. Secondly, these sequences are input into a lightweight multi-task neural network model. This model, through its lightweight design, reduces computational resource consumption, making real-time and efficient anomaly recognition possible on mobile devices, avoiding the problems of high computational load and difficulty in real-time deployment associated with traditional models. Simultaneously, the multi-task learning architecture enables the model to identify various types of anomalies in parallel and collaboratively, such as occluded areas and areas with abnormal illumination, improving the accuracy and efficiency of recognition. Finally, the model directly outputs anomaly recognition results including occluded areas and / or areas with abnormal lighting. This can directly guide the subsequent feature selection process, effectively eliminating false features caused by occlusion or abnormal lighting. This avoids the interference of false features on the pose estimation of the self-moving device, significantly improving the robustness and accuracy of the visual positioning system. Especially when service robots such as outdoor lawnmowers face complex and ever-changing environmental challenges, it can ensure stable system operation, reduce human intervention, and improve work efficiency and user experience. Step S203: Extract effective features based on multi-frame environmental images to obtain the first effective feature set. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0041] Step S204: Based on the anomaly identification results, the first effective feature set is filtered to remove false features corresponding to image anomalies, thereby obtaining the second effective feature set after anomaly removal. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0042] Step S205: Calculate the pose information of the self-moving device based on the second effective feature set. For details, please refer to [link to relevant documentation]. Figure 1Step S105 of the illustrated embodiment will not be described again here.

[0043] In some embodiments, this application further proposes a method after the lightweight multi-task neural network model outputs an anomaly recognition result of the environmental image, comprising: Step a1: Based on the occlusion area and / or the area of ​​abnormal illumination, determine the severely occluded image and the non-severely occluded image in the color image sequence.

[0044] Specifically, this step is used to classify the input color images to identify image frames that may negatively impact subsequent visual localization due to severe occlusion or abnormal lighting. Based on the anomaly recognition results output by a lightweight multi-task neural network model, the area ratio or severity score of occluded or abnormally lit regions in each image frame can be calculated. For example, if the number of pixels in the occluded region exceeds a preset percentage (e.g., 20%) of the total image pixels, or the average brightness value of the abnormally lit region deviates from a preset threshold within the normal range, the image frame is classified as a severely occluded image; otherwise, it is classified as a non-severely occluded image. Alternatively, a binary classifier can be trained, which takes the anomaly recognition results as input and directly outputs a judgment on whether an image frame is a severely occluded or non-severely occluded image.

[0045] Step a2: Remove severely occluded images from the color image sequence and remove the target grayscale image corresponding to the severely occluded image from the grayscale image sequence.

[0046] Specifically, this step physically removes frames identified as severely occluded and their corresponding grayscale images to ensure that subsequent feature extraction and pose calculation processes are not interfered with by these low-quality images. One implementation is to store the image sequence as a list or queue structure. When a severely occluded image is identified, the color image frame and its corresponding grayscale image frame with its timestamp or index in the grayscale image sequence are directly deleted from the list. Another implementation is to maintain a list of valid image frame indices or a Boolean tag array instead of directly deleting image data. Only the indices or tags of non-severely occluded images are set to valid, and subsequent processing modules only access these marked valid image frames.

[0047] Step a3: Determine whether the preset number of consecutive frames is met after removing severely occluded images; if the preset number of consecutive frames is met, output an alarm message to notify manual processing; if the preset number of consecutive frames is not met, obtain the corresponding effective color image sequence and effective grayscale image sequence based on the removed color image sequence and the removed grayscale image sequence.

[0048] Specifically, this step assesses whether the remaining valid image sequence still maintains sufficient continuity after the culling operation to support the stable operation of the visual positioning system. Too many consecutively culled image frames may lead to tracking loss. A counter can be maintained to record the number of consecutively culled, severely occluded image frames. If this counter reaches or exceeds a preset frame count condition (e.g., 5 consecutive frames), the condition is considered met. Alternatively, the proportion of valid image frames can be calculated within a sliding time window. If this proportion is lower than a preset threshold, or if the number of valid frames within the window is lower than the preset frame count condition, the condition is considered met.

[0049] If a preset number of consecutive frames are met, an alarm message is output to notify manual intervention. One implementation method is to display the warning message through the user interface of the mobile device, such as by popping up a prompt box on the screen, changing the color of an indicator light, and emitting a buzzer sound. Alternatively, the alarm message can be sent to a remote monitoring center or the operator's mobile device via a wireless communication module, such as by sending a text message, email, or push notification.

[0050] If the consecutive preset frame count condition is not met, and a corresponding valid color image sequence and a valid grayscale image sequence are obtained based on the removed color image sequence and the removed grayscale image sequence, this step is used to provide cleaned and filtered high-quality image input for the subsequent visual positioning module. One implementation is to reorganize the remaining color image frames and grayscale image frames after the removal operation into a new, corresponding image sequence as input for subsequent processing. If an index list or Boolean tag array is used for removal, these index lists or tag arrays are directly passed to subsequent modules, instructing them to process only the image frames marked as valid, thus forming a logically valid image sequence.

[0051] It is understood that the technical solution in this embodiment can dynamically filter image sequences and handle abnormal situations based on continuous frame conditions, solving the problems of false feature contamination and system failure that may occur if images are used directly after anomaly identification, thus improving the stability and reliability of visual positioning. Specifically, by determining severely occluded images and non-severely occluded images based on occlusion areas and / or abnormal lighting areas, it is possible to accurately distinguish image quality levels and avoid missing severely abnormal images. Severely occluded images are removed from the color image sequence, and corresponding target grayscale images are removed from the grayscale image sequence to ensure that invalid images are completely removed and prevent them from contaminating the subsequent feature extraction process. It is determined whether the condition of a continuous preset number of frames is met after removal, which can monitor the continuity of the image sequence and prevent the tracking thread from being interrupted due to the removal of too many images. If the condition is met, an alarm message is output to trigger manual intervention in a timely manner to prevent the system from continuing to run in an invalid state. If the condition is not met, a valid color and grayscale image sequence is obtained from the removed sequence, providing clean and corresponding input for subsequent pose calculation to ensure positioning accuracy. The above steps work together to solve key defects in abnormal image processing, enhance the adaptability of self-moving devices in complex outdoor environments, effectively avoid pose estimation errors and system robustness degradation caused by image quality issues, and improve the operating efficiency and user experience of self-moving devices in complex environments. In some embodiments, this application further proposes a visual positioning method, such as Figure 3 As shown, the method includes: Step S301: Obtain multi-frame environmental images containing time-series relationships. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0052] Step S302 involves analyzing multiple frames of environmental images to obtain anomaly identification results. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0053] Step S3021 involves preprocessing the multiple frames of environmental images to obtain corresponding color image sequences and grayscale image sequences. For details, please refer to [link to relevant documentation]. Figure 2 Step S2021 of the illustrated embodiment will not be described again here.

[0054] Step S3022: Input the color image sequence and grayscale image sequence into the lightweight multi-task neural network model. See details below. Figure 2 Step S2022 of the illustrated embodiment will not be described again here.

[0055] Step S3023: The lightweight multi-task neural network model outputs anomaly recognition results for the environmental image. These results include at least occluded areas and / or areas with abnormal lighting. For details, please refer to [link to relevant documentation]. Figure 2 Step S2023 of the illustrated embodiment will not be described again here.

[0056] Step S303: Extract effective features based on multi-frame environmental images to obtain the first effective feature set. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0057] Step S304: Based on the anomaly identification results, the first effective feature set is filtered to remove false features corresponding to image anomalies, thereby obtaining the second effective feature set after anomaly removal. Specifically, this includes: Step S3041: Determine the occlusion category of the current frame color image. The occlusion categories include slight occlusion and no occlusion.

[0058] Specifically, this step provides contextual information for subsequent feature point analysis and filtering based on the actual occlusion situation of the current image. Occlusion category determination can be achieved in several ways. For example, a pre-trained image classification model can be used to analyze color images. This model can identify the presence and degree of occlusion in the image and classify it as either slightly occluded or unoccluded. Another approach is to combine the aforementioned anomaly identification results (such as occluded areas and / or areas with abnormal lighting) and quantify the degree of occlusion by calculating the area ratio or distribution of abnormal areas, thereby determining the occlusion category.

[0059] Step S3042: Analyze the suspicious visual feature points and valid visual feature points in the first valid feature set according to the occlusion category.

[0060] Specifically, this step involves preliminarily classifying the first set of valid features extracted from multi-frame environmental images based on the determined occlusion categories, distinguishing between "suspicious" feature points that may be affected by anomalies and relatively reliable "valid" feature points. One implementation involves pre-defining image regions where anomalies may occur under different occlusion categories (e.g., image edges, specific texture regions, or regions overlapping with anomaly identification results), and then checking whether feature points in the first set of valid features fall into these regions. Feature points falling into anomalous regions are marked as suspicious visual feature points, while feature points not falling into anomalous regions are marked as valid visual feature points. Another implementation involves using a machine learning classifier, taking the feature point's descriptor, location information, and current occlusion category as features, and outputting the probability of whether the feature point is suspicious or valid, thus performing classification.

[0061] Step S3043: Add the effective visual feature points to the second effective feature set.

[0062] Specifically, this step is used to directly retain visual feature points that were identified as reliable in the initial analysis. These effective visual feature points are typically located in areas of the image that are not affected by occlusion or abnormal lighting, exhibiting high stability and contributing more positively to pose estimation. By directly adding them to the second set of effective features, it ensures that the system can fully utilize this high-quality feature information during subsequent pose calculations, avoiding the loss of valuable data due to over-filtering.

[0063] Step S3044: Effectively confirm the suspicious visual feature points and add the confirmed suspicious visual feature points to the second effective feature set.

[0064] Specifically, the purpose of this step is to further verify visual feature points initially judged as "suspicious" to distinguish between genuine spurious feature points and genuine feature points that were misjudged due to a conservative initial judgment strategy. One approach is to perform multi-frame tracking on suspicious visual feature points and combine this with a consistency check using geometric constraints (such as epipolar geometry). If a feature point exhibits geometric behavior consistent with the camera motion model across multiple consecutive frames, it is considered valid. Another approach is to use a deep learning model to analyze image patches surrounding the suspicious feature points to determine whether they are genuine environmental feature points or spurious feature points caused by occlusion.

[0065] Visual feature points that have undergone a valid verification process and are determined to be genuine are added to the second set of valid features. This not only preserves the initial valid visual feature points but also salvages those that were initially flagged as suspicious but are actually genuine. This helps maximize the use of effective information in the image while ensuring the high reliability of feature points in the second set, thus providing a more comprehensive and accurate data foundation for subsequent pose calculations.

[0066] Step S305: Calculate the pose information of the self-moving device based on the second effective feature set. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0067] It is understood that the technical solution in this embodiment, based on the analysis of multiple frames of environmental images and the acquisition of anomaly recognition results, further refines the feature point screening mechanism. By determining the occlusion category of the current frame's color image, the system can selectively classify the feature points in the first effective feature set according to the actual image quality, identifying suspicious visual feature points and valid visual feature points. Valid visual feature points are directly retained, ensuring the timely use of reliable information. For suspicious visual feature points, an effective confirmation step is introduced to accurately distinguish between genuine false feature points and misjudged genuine feature points. This effectively avoids pose calculation errors caused by inaccurate identification of false feature points in cases of slight occlusion or no occlusion. Finally, the constructed second effective feature set contains high-quality feature points that have undergone rigorous screening and confirmation, significantly improving the accuracy and robustness of pose calculation for the self-moving device. Especially in complex and changing outdoor environments, it can effectively cope with the challenges brought by instantaneous lens occlusion and changes in lighting, reducing positioning drift or failure caused by feature point contamination, thereby ensuring the stable operation and operational efficiency of the self-moving device. In some embodiments, this application further proposes analyzing suspicious visual feature points and valid visual feature points in a first valid feature set based on occlusion category, specifically including: Step b1: If the occlusion category is the unoccluded category, then visual feature points located in the non-interest region in the current frame grayscale image are taken as suspicious visual feature points, and visual feature points located in the interest region in the current frame grayscale image are taken as valid visual feature points.

[0068] The "no occlusion category" refers to a situation where no obvious occlusion is detected in the current frame's environmental image, or the degree of occlusion is extremely low and insufficient to significantly affect visual localization. This category is typically determined based on the aforementioned anomaly identification results. For example, if the area, density, or intensity of the occluded region is below a preset threshold, it can be classified as no occlusion. This can be achieved by statistically analyzing the area of ​​the occluded region output by a lightweight multi-task neural network model; if the statistical area is zero or less than a very small preset value, it is considered no occlusion. Another approach is to analyze the global texture uniformity or brightness distribution of the image. If the image as a whole exhibits good uniformity and brightness, it can be inferred to be in an occlusion-free state.

[0069] In visual localization tasks, "regions of non-interest" (ROIs) refer to image regions that typically lack stable, reliable feature points or are easily affected by environmental interference. For example, for outdoor self-mounted devices, the sky, blurred distant views, or certain areas known to be susceptible to lighting conditions can be defined as RIOs. Marking visual feature points in these regions as suspicious is to prevent potentially low-quality or unstable feature points from interfering with pose estimation. One approach is to predefine one or more image region masks; for example, using a semantic segmentation model to identify regions such as the sky and water surfaces and label them as RIOs. Another approach is to classify regions containing feature points that are too far from the camera or whose motion patterns do not conform to expectations as RIOs based on the depth information or motion patterns of the feature points.

[0070] In visual localization tasks, a "region of interest" (ROI) is an image region expected to contain a large number of stable and reliable feature points. For example, for outdoor self-operated mobile devices, areas such as the ground, grass, and obstacle edges are typically considered RIOs. Directly labeling the visual feature points of these regions as valid is based on the fact that these feature points generally have high reliability and stability, providing accurate information for pose estimation. One implementation method is to determine the RIO in the opposite way to the definition of a non-ROI; that is, the portion of the image other than the non-ROI is considered the RIO. Another implementation method is to use a pre-defined image region (e.g., the center or bottom of the image) or a specific target region identified by an object detection algorithm as the RIO.

[0071] Step b2: If the occlusion category is slight occlusion, then perform anomaly region identification in the region of interest of the current frame grayscale image; The "slight occlusion category" refers to a situation where there is some degree of occlusion in the current frame's environmental image, but it has not yet reached the point of severely affecting visual localization. Examples include a small amount of water droplets, grass clippings, or localized glare on the lens. This category is typically determined based on anomaly detection results. When the area, density, or intensity of the occluded region falls between two preset thresholds, it can be classified as a slight occlusion. One approach is to statistically analyze the area of ​​the occluded region output by a lightweight multi-task neural network model. If the statistically analyzed area is greater than zero and less than a preset severe occlusion threshold, it is classified as slight occlusion. Another approach is to analyze the local contrast or texture features of the image. If a local area shows anomalies but the overall image structure remains intact, it can be inferred to be a state of slight occlusion.

[0072] Specifically, this step aims to further refine the detection of potential local anomalies within the region of interest (ROI) of an image, especially in cases of slight occlusion. These anomalies may be caused by minor lens attachments, localized lighting anomalies (such as glare or bright spots), or dynamic disturbances in the environment. While they may not constitute large-area occlusion, their internal feature points can still be misleading. The purpose of anomaly region identification is to more accurately eliminate these potential false features. One approach is to utilize image segmentation techniques, such as deep learning-based semantic segmentation models, to perform pixel-level anomaly detection within the ROI, identifying regions that do not conform to the normal environmental texture. Another approach is to employ traditional image processing algorithms, such as local contrast analysis, edge detection, or optical flow analysis, to detect regions within the ROI that exhibit abnormal local brightness, color, or motion patterns, and mark them as anomalies.

[0073] Step b3: Visual feature points located in non-interest regions and abnormal regions in the current frame grayscale image are designated as suspicious visual feature points, and visual feature points located in the interest region but not in the abnormal region are designated as valid visual feature points.

[0074] Specifically, this step uniformly marks all visual feature points within regions deemed unreliable (including regions of non-interest and anomalous regions identified under slight occlusion) as suspicious. The aim is to ensure that all feature points potentially introducing errors are included in the subsequent validity verification process, thereby minimizing the interference of false features on pose estimation. One implementation is to merge the masks of the regions of non-interest and anomalous regions using a logical OR operation to form a single suspicious region mask, and then mark all feature points falling within this mask as suspicious. Another implementation is to iterate through the coordinates of each feature point after feature point extraction, determining whether it lies within a region of non-interest or an anomalous region; if so, it is classified as a suspicious visual feature point.

[0075] Visual feature points located within the region of interest (ROI) but not within anomaly regions are considered valid visual feature points. Those visual feature points located within the core of the image, generally reliable ROIs, and confirmed not to belong to anomaly regions after anomaly region identification, are directly marked as valid. These feature points are considered stable and reliable features in the current frame and can be directly used for pose calculation. This classification method ensures that only feature points that have undergone double screening (first, the ROI, then exclusion of anomaly regions) are considered valid, thus guaranteeing the purity of the feature set used for pose estimation. One implementation is to combine the ROI mask and the anomaly region mask using logical NAND operations to obtain a "ROI and non-anomaly" region mask, and then mark all feature points falling within this mask as valid. Another implementation is to determine whether a feature point is located within the ROI and simultaneously whether it is not located within anomaly regions during feature point traversal; if both conditions are met, it is classified as a valid visual feature point.

[0076] It is understood that the technical solution in this embodiment can perform more refined classification and screening of visual feature points in the first effective feature set according to different occlusion categories. In the absence of occlusion, the system can efficiently identify feature points in non-interest regions as suspicious, while directly considering feature points in the interest region as valid, thus ensuring efficiency while avoiding interference from low-quality feature points. In the case of slight occlusion, this application further identifies abnormal regions within the interest region, enabling the system to accurately detect and isolate false feature points caused by partial occlusion, abnormal lighting, etc. By uniformly classifying feature points in non-interest regions and abnormal regions as suspicious, and only considering feature points in the non-abnormal parts of the interest region as valid, the problem of accurately distinguishing feature points in non-interest regions and abnormal regions under different occlusion categories to avoid the failure to identify false features is effectively solved. This improves the purity and reliability of the second effective feature set. In some embodiments, this application further proposes that, before identifying visual feature points located in non-interest regions and abnormal regions in the current frame grayscale image as suspicious visual feature points, the method further includes: Step c1: Calculate the abnormal area of ​​the abnormal region.

[0077] Specifically, the statistical analysis of anomalous areas refers to the quantification of image regions that may contain anomalies (such as stains, water droplets, or slight occlusions) within the region of interest of the current frame's grayscale image under the category of slight occlusion. The statistical analysis of anomalous areas can be achieved in several ways. For example, the number of pixels contained in the anomalous region can be calculated; if the anomalous region is represented by a binary mask, the total number of pixels with a value of 1 in the mask can be counted. Alternatively, the anomalous area can be approximated by geometrically fitting the anomalous region (such as using a minimum bounding rectangle or convex hull) and then calculating the area of ​​the fitted region.

[0078] Step c2: If the abnormal area is greater than the preset area, discard the current frame grayscale image; if the abnormal area is less than or equal to the preset area, execute the step of taking the visual feature points in the current frame grayscale image located in the non-interest area and the abnormal area as suspicious visual feature points.

[0079] The preset area is a threshold used to determine whether anomaly regions are excessively large. Its function is to set an acceptable upper limit for the degree of anomaly, avoiding the processing of image frames that might introduce a large number of false features due to excessively large anomaly areas. This preset area can be set through experimental testing and empirical values, based on the actual application scenario, the camera parameters of the mobile device, and the requirements for visual positioning accuracy and robustness. For example, it can be set as a percentage of the total image area or a fixed pixel area value. Furthermore, the preset area can also be dynamically adjusted, for example, calculated in real-time using machine learning models or adaptive algorithms based on factors such as ambient lighting and the speed of the mobile device.

[0080] When the statistically obtained abnormal area exceeds a preset area, it indicates a high degree of abnormality in the current frame image, suggesting that most of its feature points may be false or contaminated. Discarding the current frame's grayscale image in this case prevents this highly contaminated data from entering subsequent feature filtering and pose calculation processes, thus preventing false features from severely impacting visual positioning accuracy. Discarding can manifest as directly skipping subsequent processing of the current frame, not extracting any feature points from it, and not using it for pose calculation; alternatively, it can mark the frame as invalid and trigger an alarm or attempt to restart processing from the next frame to maintain system continuity.

[0081] Conversely, if the abnormal area is less than or equal to a preset area, it indicates that the abnormality of the current frame image is within an acceptable range. The system will continue to execute the step of classifying visual feature points located in non-interest regions and abnormal regions in the current frame grayscale image as suspicious visual feature points. This step ensures that even with slight occlusion and mild abnormalities, the system can continue to extract and classify feature points from the image, maintaining the continuity of visual localization. The specific execution method of this step is consistent with that described in the above scheme, that is, marking the feature point as a suspicious visual feature point based on the region where the feature point is located (non-interest region or abnormal region). On this basis, feature points within the abnormal region can be further weighted or assigned a lower confidence level for more stringent screening in the subsequent effective confirmation stage.

[0082] It is understood that the technical solution in this embodiment optimizes the feature point screening process under slight occlusion by introducing abnormal area statistics and a preset area comparison mechanism, effectively solving the problem that false features may be introduced when the abnormal area is too large. In some embodiments, this application further proposes an effective method for identifying suspicious visual feature points, the identification process including: Step d1: Track suspicious visual feature points in the preset continuous frame images to obtain the tracking position of the suspicious visual feature points in each frame image.

[0083] Specifically, this step is used to obtain the actual motion trajectory of suspicious visual feature points over time for subsequent effectiveness evaluation. One implementation is to use an optical flow-based method, such as the KLT (Kanade-Lucas-Tomasi) optical flow algorithm. This method estimates the motion of feature points by analyzing changes in pixel intensity in an image sequence, thereby tracking suspicious visual feature points across consecutive frames and obtaining their precise pixel coordinates in each frame. Another implementation is to use a descriptor-based matching method. First, local descriptors of suspicious visual feature points are extracted from consecutive frames. Then, the correspondence between feature points in different frames is established by calculating the similarity between descriptors, thus determining their tracking position.

[0084] Step d2: Calculate the camera pose transformation matrix of the preset continuous frame images using effective visual feature points.

[0085] Specifically, the purpose of this step is to accurately estimate the camera motion between consecutive frames based on confirmed and reliable valid visual feature points, thereby providing an accurate geometric basis for subsequently predicting the theoretical positions of suspicious visual feature points. By using only valid visual feature points, the contamination of camera pose estimation by suspicious feature points can be avoided. One implementation is to use the Perspective-n-Point (PnP) algorithm combined with robust estimation methods such as RANSAC (Random Sample Consensus). This method uses the 2D image coordinates of valid visual feature points and their corresponding 3D world coordinates (if known, or obtained through triangulation) to solve for the camera pose (rotation and translation), and uses RANSAC to eliminate possible outlier matches, ensuring the accuracy of pose estimation. Another implementation is to use a least-squares-based optimization method. By constructing an optimization problem to minimize the reprojection error of valid visual feature points between different frames, the optimal camera pose transformation matrix is ​​iteratively solved.

[0086] Step d3: Use the camera pose transformation matrix to predict the theoretical position of suspicious visual feature points from the previous frame to the next frame.

[0087] Specifically, this step is used to infer the expected location of a suspicious visual feature point in the next frame based on the known motion of the camera and the geometric constraints of the scene. One implementation involves first inverting the 2D image coordinates of the suspicious visual feature point in the current frame using a back-projection operation, combined with the camera parameters of the current frame, to obtain its 3D point in the current camera coordinate system. Then, using the calculated camera pose transformation matrix, the 3D point is transformed from the current camera coordinate system to the camera coordinate system of the next frame. Finally, the transformed 3D point is projected back to the 2D image plane of the next frame to obtain its theoretical location. Another implementation involves, if the suspicious visual feature point has been successfully triangulated and its 3D map point in the world coordinate system has been obtained, directly projecting the 3D map point onto the image plane of the next frame using the camera pose transformation matrix to obtain its theoretical location.

[0088] Step d4: Determine whether the suspicious visual feature points are valid based on the reprojection error between the tracked position and the theoretical position in each frame.

[0089] Specifically, the difference between the actual and predicted motion of suspicious visual feature points is quantified to distinguish between real environmental features and false interference. One approach is to calculate Euclidean distance. The pixel distance between the tracked position (actual observation) and the theoretical position (predicted) of a suspicious visual feature point in each frame is directly calculated. If this distance exceeds a preset threshold, the feature point is considered invalid; otherwise, it is considered valid. Another approach is to calculate Mahalanobis distance. Considering the uncertainty in feature point tracking or localization, Mahalanobis distance is used to measure the difference between the tracked position and the theoretical position, which can more robustly handle noise and uncertainty. The validity of the feature point is determined by comparing the magnitude of the Mahalanobis distance with a preset threshold.

[0090] It is understood that the technical solution in this embodiment provides an effective verification mechanism based on reprojection error, which can reliably verify the validity of suspicious visual feature points. By tracking suspicious visual feature points in a preset series of images, their actual motion trajectory in the time series is obtained, avoiding misjudgments that may occur based on information from only a single frame. Simultaneously, the camera pose transformation matrix is ​​calculated using the confirmed valid visual feature points, effectively preventing the contamination of pose calculation by suspicious feature points. The theoretical position of suspicious visual feature points from the previous frame to the next frame is predicted using the accurate camera pose transformation matrix, providing a reliable geometric benchmark for evaluating their validity. Finally, by comparing the reprojection error between the tracked position and the theoretical position of the suspicious visual feature points in each frame, the real environmental features and false feature points caused by occlusion, abnormal lighting, etc., can be quantitatively and objectively distinguished. This effectively reduces the interference of false feature points on the pose estimation of the self-moving device, improving the accuracy and stability of the visual positioning system. In some embodiments, this application further proposes determining whether a suspicious visual feature point is valid based on the reprojection error between the tracked position and the theoretical position in each frame, specifically including: Step e1: Calculate the average reprojection error based on all reprojection errors.

[0091] Reprojection error refers to the pixel distance between the projected position of a feature point in 3D space onto the image plane and the actual observed position of the feature point. After tracking suspicious visual feature points and predicting their theoretical positions in a set of consecutive frames, a series of reprojection errors can be obtained. The average reprojection error can be calculated by arithmetically averaging the reprojection errors of all suspicious visual feature points in the set of consecutive frames, or by using a weighted average method. For example, different weights can be assigned to the errors of different frames based on tracking stability or image quality to obtain a more representative error metric. Averaging helps smooth out random fluctuations in single-frame errors, providing a more stable basis for judgment.

[0092] Step e2: When the average reprojection error is less than or equal to the first preset error threshold, the suspicious visual feature point is determined to be invalid.

[0093] The first preset error threshold is a small value, such as 1 to 2 pixels. When the average reprojection error is very small, even close to zero, it may mean that the "feature point" is actually a static interference such as dirt, water droplets, or other attachments on the lens. These are fixed relative to the camera, so their positional changes on the image do not match the projection changes of environmental feature points when the camera moves, resulting in a very small reprojection error. Deeming them invalid can prevent these false feature points from being mistaken for environmental features, thus preventing them from contaminating the second set of valid features.

[0094] Step e3: When the average reprojection error is greater than the first preset error threshold and less than or equal to the second preset error threshold, the suspicious visual feature point is determined to be valid.

[0095] The second preset error threshold is a relatively large value, for example, it can be set to 5 to 10 pixels. When the average reprojection error is between the first and second preset error thresholds, it indicates that the suspicious visual feature point has a certain reprojection error during tracking, but this error is within an acceptable range, consistent with the projection error characteristics that real-world feature points may produce when the camera moves. Judging it as valid means that the feature point is a stable feature in the real environment and can be included in the second set of valid features for subsequent pose calculations.

[0096] Step e4: When the average reprojection error is greater than the second preset error threshold, the suspicious visual feature points are discarded.

[0097] When the average reprojection error exceeds a second preset error threshold, it typically indicates that the suspicious visual feature point exhibits significant instability during tracking, with its positional changes on the image deviating severely from the theoretically predicted position. This could be due to the feature point being a dynamic object, tracking failure, or a highly unstable spurious feature. Discarding it effectively prevents these high-error feature points from introducing significant noise and errors into pose estimation, thereby ensuring the accuracy and robustness of pose calculation.

[0098] It is understood that the technical solution in this embodiment introduces an average reprojection error based on multi-frame tracking results when effectively confirming suspicious visual feature points. This effectively smooths out the fluctuations in single-frame errors caused by factors such as environmental noise, changes in illumination, or temporary occlusion, making the judgment of the validity of feature points more stable and reliable. In some embodiments, this application further proposes calculating the pose information of a self-moving device based on a second set of valid features, including: Step f1: Match the second set of valid features of the current image with the map points of the previous frame image to obtain the pose information of the self-moving device.

[0099] The process involves matching the second set of valid features of the current image with map points from the previous frame. The aim is to establish a correspondence between reliable visual feature points selected in the current frame and the map points constructed in the previous frame. This links the current observation with the known environmental structure. Matching can be achieved in various ways. For example, feature descriptor-based methods can be used, such as feature extraction algorithms like ORB (Oriented Fast and Rotated BRIEF), SIFT (Scale-Invariant Feature Transform), or SuperPoint, to generate a unique descriptor for each feature point in the second set of valid features. Then, by calculating the similarity between descriptors (such as Hamming distance or Euclidean distance), the best matching pair between the feature points in the current frame and the map points in the previous frame is found. To improve matching accuracy, geometric constraints (such as the RANSAC algorithm) can be incorporated to eliminate incorrect matching pairs. Another approach is to use learning-based methods, such as deep learning models (like SuperGlue), to directly predict the matching relationships between feature points, better handling scenarios with sparse feature points or significant appearance variations. In addition, optical flow can be used to track feature points and associate the tracked feature points with map points.

[0100] After obtaining the matching relationship between the second effective feature set of the current image and the map points of the previous frame image, the pose information of the self-moving device can be obtained. Acquiring pose information is crucial for achieving autonomous navigation and localization of the self-moving device. The pose of the self-moving device can be calculated by solving the Perspective-n-Point (PnP) problem. Furthermore, more complex optimization methods, such as local or global Bundle Adjustment (BA), can be employed to jointly optimize the matching results and historical poses to obtain more accurate pose estimation. In some cases, information from other sensors, such as the Inertial Measurement Unit (IMU), can be fused, and methods such as Kalman filtering or extended Kalman filtering can be used to further improve the robustness and accuracy of pose estimation.

[0101] It is understood that the technical solution in this embodiment, by matching the second effective feature set of the current image with the map points of the previous frame image, can effectively solve the problem of inaccurate pose estimation or system tracking failure caused by scarce feature points or interference from false features in complex outdoor environments. This avoids system relocalization or tracking loss due to insufficient feature points. It improves the accuracy of pose estimation and enhances the robustness of the system in dynamic and complex environments, thereby ensuring the stability and reliability of self-moving device navigation.

[0102] In one example, a lawnmower is used as a self-moving device to illustrate the visual positioning method in more detail through a more concrete example: Step g1: Environmental image acquisition and preprocessing.

[0103] First, the original RGB image with timestamp t is captured using the front-facing camera of the lawnmower, denoted as I_t. Second, gamma correction and histogram equalization are performed on I_t to eliminate uneven brightness and low contrast, improving the overall image quality. Then, the enhanced image is scaled and cropped (to match the input size requirements of subsequent models) and converted to grayscale, outputting an RGB image (denoted as Ir_t) for image classification and segmentation tasks, and a grayscale image (denoted as Ig_t) for visual feature extraction tasks.

[0104] Step g2: Classification and segmentation of environmental image occlusion and lighting anomalies.

[0105] First, the preprocessed RGB image Ir_t is input into a lightweight multi-task neural network model (based on improved YOLO and DeepLab architectures, adapted to the lightweight requirements of outdoor scenes) to perform image inference computation. This neural network model has been trained on a large-scale outdoor lawnmower camera image dataset, which includes normal images, severely occluded images (scenes with foliage cover, water droplets, mud, etc.), severely abnormal lighting images (scenes with large-area glare, overexposure, halo interference, etc.), slightly occluded images, and slightly abnormal lighting images, demonstrating accurate classification and segmentation capabilities. Then, the model outputs two core results: Image classification result Cls_t: Determines the category of the current image (normal / severe occlusion / severe lighting abnormality / slight occlusion / slight lighting abnormality); Seg_t, a probability map for segmenting abnormal regions, marks the location and extent of occlusion and lighting abnormalities in the image in terms of probability.

[0106] Step g3: Abnormal environment image classification and processing and multi-scale visual feature extraction.

[0107] If Cls_t determines that the image is "severely occluded" or "severely abnormal lighting", the subsequent routine processing flow of the image will be terminated and the abnormal information will be fed back to the "system status monitoring and adaptive module". If Cls_t is determined to be "normal", "slightly occluded", or "slightly abnormal lighting", the grayscale image Ig_t with timestamp t is sent to the visual feature extraction module.

[0108] If a single frame or a few frames are abnormal, the current frame is skipped and the image processing flow of the next frame is started. At the same time, the SLAM system is controlled to perform a degradation smoothing operation (such as reducing the weight of visual data) to ensure the system’s rapid response and stability to sudden anomalies. If multiple consecutive frames of images show severe anomalies, and the situation does not improve after implementing specified remedial measures (such as reversing the lawnmower or rotating the vehicle to adjust the shooting angle), the module will trigger an alarm (such as pausing the equipment or providing an audio-visual alert).

[0109] For the filtered Ig_t image, key points and corresponding descriptors are calculated and extracted across the entire image. After filtering, a multi-scale visual feature point set is obtained, denoted as F_total={f1,f2,f3,...,f n (n is the number of feature points).

[0110] Step g4: Filter out false features in the environmental image.

[0111] First, the segmentation probability map Seg_t output in step g2 is binarized to obtain a mask image Mask_t with the same resolution. In this image, a pixel value of 255 (white) represents a suspected occlusion / illumination abnormality area, and a pixel value of 0 (black) represents a clean area without abnormalities. Secondly, the ROI region is defined. ROI_t is set as a manually or dynamically defined region of interest in the image. A pixel value of 255 (white) represents an effective region (such as buildings, vegetation, objects, etc., which are rich in effective features), and a pixel value of 0 (black) represents an invalid region (such as the sky, water, etc., which are prone to generating invalid features). ROI_t can be set as a fixed region, or a dynamic region can be generated by a dedicated semantic segmentation model to reduce interference from invalid features.

[0112] For images with "slight occlusion / slight lighting anomalies", the coordinates (u,v) of each feature point f_i are compared with the corresponding pixel values ​​of Mask_t and ROI_t. If Mask_t(u,v)=255 (located in the abnormal region) or ROI_t(u,v)=0 (located in the non-ROI region), f_i is marked as a suspicious feature point and added to the set F_suspect; if Mask_t(u,v)=0 and ROI_t(u,v)=255, f_i is marked as a reliable feature point and added to the set F_reliable. For normal images, only the coordinates of feature points are compared with ROI_t. If ROI_t(u,v)=0, f_i is marked as a suspicious feature point (included in F_suspect); if ROI_t(u,v)=255, f_i is marked as a reliable feature point (included in F_reliable). This reduces the amount of computation in normal scenarios and improves the real-time performance of the system.

[0113] Next, suspicious feature points are tracked. Specifically, in the subsequent N consecutive frames (N=5 recommended), feature points in F_suspect are tracked using optical flow or feature matching. Using reliable feature points in F_reliable, a high-confidence camera pose transformation matrix T_(t→t+1) (pose transformation from frame t to frame t+1) is calculated using the PnP (PerspectivenPoint) algorithm or essential matrix factorization. Based on the coordinates of the suspicious feature points in the previous frame and T_(t→t+1), their coordinates (u_pred, v_pred) in the current frame are predicted. The Euclidean distance between the predicted coordinates and the actual coordinates (u_actual, v_actual) is calculated as the reprojection error. If the average error of N consecutive frames is less than the threshold θ1 (very low threshold), it indicates that the feature point motion is highly consistent with the camera motion model, and it is judged as a "false feature point" (an occlusion feature fixed to the camera), classified into F_fake and filtered out. If θ1 ≤ average error < threshold θ2 (a larger threshold), it indicates that the feature point may be a dynamic object or a tracking deviation, and it should be handled according to the original SLAM mechanism (such as RANSAC algorithm for removal). If stable tracking is not possible, the feature point should be discarded directly. The final feature set used for SLAM is F_pure = F_reliable + (F_suspect + F_fake) (the set of reliable feature points and the filtered suspicious feature points).

[0114] Step g5: SLAM processing based on the second effective feature.

[0115] First, the F_pure is input into the standard visual SLAM process. By matching the F_pure with map points from the previous frame, the precise pose of the current camera is calculated. Second, new map points are generated using the new keyframe and feature points from the F_pure, and Bundle Adjustment is performed on the local map to eliminate local pose errors. Finally, locations visited by the lawnmower are identified, triggering global pose map optimization to eliminate accumulated errors from long-term operation. Due to the high purity of F_pure features, the false matching rate of loop closure detection is significantly reduced, while avoiding the generation of "ghost" map points (invalid false map points), ensuring map accuracy.

[0116] Step g6: Self-moving device status monitoring and adaptation.

[0117] In real-time monitoring step g4, the "proportion of false feature points" (size(F_fake) / size(F_total)) is monitored. If this proportion continues to be higher than the alarm threshold (30% recommended), it indicates that lens occlusion / abnormal lighting has affected SLAM accuracy. The system sends a warning message to the user through the communication module: "Lens is dirty, cleaning is recommended".

[0118] If a single frame is severely abnormal, frame skipping is performed to temporarily reduce the dependence on visual SLAM, increase the data fusion weight of wheel odometry and IMU (inertial measurement unit), and achieve smooth system degradation.

[0119] If the severe anomaly persists, the lawnmower will be controlled to pause operation or automatically return to the base station to avoid operational deviations or collision risks caused by insufficient SLAM accuracy.

[0120] This embodiment also provides a visual positioning device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0121] This embodiment provides a visual positioning device, such as... Figure 4 As shown, it includes: The acquisition module 401 is used to acquire multi-frame environmental images containing time-series relationships.

[0122] The recognition module 402 is used to analyze multiple frames of environmental images to obtain anomaly recognition results in the environmental images.

[0123] The extraction module 403 is used to extract effective features based on multi-frame environmental images to obtain a first set of effective features.

[0124] The elimination module 404 is used to filter the first effective feature set according to the anomaly recognition result, eliminate false features corresponding to image anomalies, and obtain the second effective feature set after anomaly elimination.

[0125] The calculation module 405 is used to calculate the pose information of the self-moving device based on the second effective feature set.

[0126] In some embodiments, this application also discloses a self-moving device, such as... Figure 5As shown, it includes: an image acquisition component, a motion component, a power component, a memory, and a processor. The image acquisition component is used to acquire environmental images, the motion component is used to record the pose of the self-moving device and perform self-movement, the power component is used to provide power to the self-moving device, and the memory and processor are interconnected. The memory stores computer instructions, and the processor executes the above-mentioned visual positioning method by executing the computer instructions.

[0127] Specifically, by dynamically and collaboratively combining the image acquisition component, motion component, power component, memory, and processor, hardware-level support is provided for anomaly recognition and feature purification of multi-frame environmental images. This effectively suppresses false feature contamination caused by lens occlusions and lighting changes, ensuring the reliability of pose calculation. The image acquisition component continuously acquires multi-frame environmental images with a time-series relationship, providing a temporal analysis basis for anomaly recognition; the motion component records the device pose in real time and performs self-movement operations, ensuring that the motion trajectory is synchronized with the positioning results; the power component stably supplies the energy required for device operation, avoiding power fluctuations from interfering with the continuity of positioning; the memory and processor collaboratively execute optimized visual positioning logic, where the processor performs anomaly region detection on multi-frame environmental images based on instructions in the memory, identifying image anomaly regions caused by branches, water droplets, or overexposure of strong light, and accordingly filters the initially extracted first effective feature set, removing false feature points located in the anomaly regions, and finally uses the purified second effective feature set to calculate pose information.

[0128] The visual positioning apparatus provided in this application can execute the visual positioning method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0129] Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. See below for details. Figure 6 This diagram illustrates a suitable structural schematic for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0130] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0131] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the visual positioning method of embodiments of this application.

[0132] Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0133] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the visual positioning method shown in the above embodiments is implemented.

[0134] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0135] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A visual positioning method applied to a self-moving device, characterized in that, The method includes: Acquire multi-frame environmental images containing time-series relationships; Analyze multiple frames of the environmental images to obtain anomaly identification results in the environmental images; Based on multiple frames of the environmental images, effective features are extracted to obtain a first set of effective features; Based on the anomaly identification results, the first effective feature set is filtered to remove false features corresponding to image anomalies, resulting in a second effective feature set after anomaly removal. Based on the second set of valid features, the pose information of the self-moving device is calculated.

2. The method of claim 1, wherein, The step of analyzing multiple frames of the environmental images to obtain anomaly identification results in the environmental images includes: The environmental images of multiple frames are preprocessed to obtain corresponding color image sequences and grayscale image sequences; The color image sequence and the grayscale image sequence are input into a lightweight multi-task neural network model; The lightweight multi-task neural network model outputs anomaly recognition results for the environmental image, and the anomaly recognition results include at least occluded areas and / or areas with abnormal lighting.

3. The method according to claim 2, characterized in that, After the lightweight multi-task neural network model outputs the anomaly recognition result of the environmental image, the method includes: Based on the occlusion area and / or the abnormal lighting area, determine the severely occluded images and non-severely occluded images in the color image sequence; Remove the severely occluded image from the color image sequence, and remove the target grayscale image corresponding to the severely occluded image from the grayscale image sequence; Determine whether the preset number of consecutive frames is met after removing the severely occluded image; If the preset number of consecutive frames is met, an alarm message is output to notify manual handling. If the condition of the number of consecutive preset frames is not met, then a corresponding effective color image sequence and an effective grayscale image sequence are obtained based on the removed color image sequence and the removed grayscale image sequence.

4. The method according to claim 3, characterized in that, The step involves filtering the first effective feature set based on the anomaly identification result, removing false features corresponding to image anomalies, and obtaining a second effective feature set after anomaly removal, including: Determine the occlusion category of the current frame's color image, where the occlusion category includes a slight occlusion category and a no-occlusion category; Based on the occlusion category, analyze the suspicious visual feature points and valid visual feature points in the first valid feature set; Add the effective visual feature points to the second effective feature set; The suspicious visual feature points are effectively confirmed, and the confirmed suspicious visual feature points are added to the second set of valid features.

5. The method according to claim 4, characterized in that, The step of analyzing the suspicious visual feature points and valid visual feature points in the first valid feature set according to the occlusion category includes: If the occlusion category is the unoccluded category, then visual feature points located in the non-interest region in the current frame grayscale image are taken as the suspicious visual feature points, and visual feature points located in the interest region in the current frame grayscale image are taken as the valid visual feature points. If the occlusion category is a slight occlusion category, then abnormal region identification is performed in the region of interest of the current frame grayscale image; Visual feature points located in the non-interest region and the abnormal region in the current frame grayscale image are designated as suspicious visual feature points, and visual feature points located in the interest region but not in the abnormal region are designated as valid visual feature points.

6. The method according to claim 5, characterized in that, Before identifying visual feature points located in the non-interest region and the abnormal region in the current frame grayscale image as the suspicious visual feature points, the method further includes: Calculate the abnormal area of ​​the aforementioned abnormal region; If the abnormal area is greater than a preset area, the current frame grayscale image is discarded; if the abnormal area is less than or equal to the preset area, the step of taking the visual feature points in the current frame grayscale image located in the non-interest region and the abnormal region as the suspicious visual feature points is executed.

7. The method according to claim 4, characterized in that, The effective confirmation of the suspicious visual feature points includes: The suspicious visual feature points are tracked in a preset series of images to obtain the tracking position of the suspicious visual feature points in each frame of the image; The camera pose transformation matrix of the preset consecutive frame images is calculated using the effective visual feature points; The theoretical position of the suspicious visual feature point from the previous frame to the next frame is predicted using the camera pose transformation matrix. The validity of the suspected visual feature point is determined based on the reprojection error between the tracked position and the theoretical position in each frame.

8. The method according to claim 7, characterized in that, The step of determining whether the suspicious visual feature point is valid based on the reprojection error between the tracked position and the theoretical position in each frame includes: The average reprojection error is calculated based on all reprojection errors. When the average reprojection error is less than or equal to the first preset error threshold, the suspicious visual feature point is determined to be invalid. When the average reprojection error is greater than the first preset error threshold and less than or equal to the second preset error threshold, the suspicious visual feature point is determined to be valid. When the average reprojection error is greater than the second preset error threshold, the suspicious visual feature points are discarded.

9. The method according to claim 1, characterized in that, The step of calculating the pose information of the self-moving device based on the second effective feature set includes: The pose information of the self-moving device is obtained by matching the second set of valid features of the current image with the map points of the previous frame image.

10. A self-moving device, characterized in that, include: The device includes an image acquisition component, a movement component, a power component, a memory, and a processor. The image acquisition component is used to acquire environmental images. The movement component is used to record the pose of a self-moving device and perform self-movement. The power component is used to provide power to the self-moving device. The memory and the processor are interconnected. The memory stores computer instructions. The processor executes the computer instructions to perform the visual positioning method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the visual positioning method according to any one of claims 1 to 9.

Images