Obstacle identification method of self-moving device, self-moving device and medium

Abstract

The invention relates to an obstacle identification method, in particular to an obstacle identification method of self-moving equipment, the self-moving equipment and a medium, and aims to solve the problems that a visual perception system of the self-moving equipment for mowing operation is easy to misjudge an interferent as an obstacle, so that unnecessary obstacle avoidance operation is triggered, and the working efficiency is improved. Even, the self-moving equipment can be stuck in a boundary area, and normal mowing task execution is affected. The obstacle recognition method of the self-moving device comprises the steps of firstly obtaining clear and stable image input, performing preliminary correction and filtering processing, then extracting non-grass and non-ground areas in the image as binary images, extracting depth information as 3D point cloud data, and screening the 3D point cloud data through the binary images to obtain obstacle 3D point cloud so as to realize obstacle recognition. According to the method, the non-grass binary image and the non-ground binary image are extracted separately, so that higher semantic definition and misrecognition tolerance are kept.

Description

Technical Field

[0001] The present invention relates to an obstacle recognition method, and in particular to an obstacle recognition method for a self-moving device, a self-moving device and a medium. Background Art

[0002] With the widespread adoption of robotic lawn mowing devices in home gardening and public greening management, the demand for autonomous navigation and obstacle avoidance capabilities is increasing. As the core module for environmental understanding and path planning, the recognition accuracy of the visual perception system directly impacts the consistency and safety of mowing tasks.

[0003] Visual perception systems face numerous challenges in real-world outdoor environments, especially during dry seasons and autumn and winter, when numerous objects with similar colors and shapes to obstacles, such as dead grass and bare earth, often appear on the ground. These objects can easily be misidentified as obstacles, triggering unnecessary obstacle avoidance maneuvers and even causing the autonomous vehicle to become stuck in border areas, disrupting the normal mowing task. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that the visual perception system of a self-moving device used for mowing operations easily misjudges interference objects as obstacles, thereby triggering unnecessary obstacle avoidance operations, and may even cause the self-moving device to get stuck in the boundary area, affecting the normal execution of the mowing task. A method for obstacle recognition of a self-moving device, a self-moving device and a medium are provided.

[0005] In order to solve the deficiencies of the above-mentioned prior art, the present invention provides the following technical solutions:

[0006] A method for identifying obstacles in a self-propelled device, which is special in that it includes the following steps:

[0007] enabling at least one camera provided on the mobile device to capture an image;

[0008] Outputting corresponding non-grass binary images, non-ground binary images, and 3D point cloud data based on the image; the 3D point cloud data includes mapping pairs consisting of image pixels and corresponding depth values, or a depth map generated from the mapping pairs;

[0009] Combining the non-grass binary map, the non-ground binary map, and the 3D point cloud data, a 3D point cloud of the obstacle is output; the 3D point cloud of the obstacle is output in a structured format that includes the three-dimensional bounding box information and type label of the obstacle, and is recorded in a local memory or sent to a navigation system or a remote server through a communication interface.

[0010] Optionally, the image is an image of continuous frames;

[0011] The step of combining the non-grass binary image, the non-ground binary image, and the 3D point cloud data to output the obstacle 3D point cloud includes:

[0012] Align and merge the non-grass binary map and the non-ground binary map to obtain the obstacle binary map;

[0013] The obstacle binary map is used to extract the obstacle area corresponding to the 3D point cloud data to obtain the obstacle 3D point cloud; the obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the three-dimensional bounding box information of the obstacle;

[0014] Then output the obstacle 3D point cloud.

[0015] Optionally, the camera includes at least one binocular camera;

[0016] Each of the binocular cameras corresponds to an IMU, and the IMU is set on the mobile device;

[0017] The images are images of continuous frames;

[0018] The step of combining the non-grass binary image, the non-ground binary image, and the 3D point cloud data to output the obstacle 3D point cloud includes:

[0019] Align and merge the non-grass binary image and non-ground binary image of each frame to obtain the obstacle binary image of each frame;

[0020] Use IMU data to correct the 3D point cloud data of each frame to obtain the corresponding corrected 3D point cloud data;

[0021] The obstacle binary map of each frame is used to extract the obstacle area in the corresponding 3D point cloud data after correction, and obtain the obstacle 3D point cloud of each frame; the obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the 3D bounding box information of the obstacle;

[0022] Then output the obstacle 3D point cloud for each frame.

[0023] Optionally, correcting each frame of 3D point cloud data using IMU data includes:

[0024] Collect IMU data from each IMU at a preset sampling frequency within a preset acquisition time.

[0025] For each IMU, independently calculate the attitude angle of each set of collected IMU data;

[0026] If the position of the mobile device corresponding to a certain IMU is determined to be on a slope based on the average attitude angle within the preset acquisition time, the attitude transformation of the corresponding frame 3D point cloud data is triggered;

[0027] The corresponding frame refers to the frame whose timestamp is closest to the center moment of the preset acquisition duration.

[0028] Optionally, before outputting the obstacle 3D point cloud of each frame, the process further includes:

[0029] Combined with RTK information, each frame of the obstacle 3D point cloud is converted to the world coordinate system to obtain the obstacle 3D point cloud in the world coordinate system. The obstacle 3D point cloud in the world coordinate system includes: the position of each point in the world coordinate system, the timestamp, and the 3D bounding box information of each obstacle;

[0030] The obstacle 3D point cloud of each frame is corrected through continuous frame detection, and the updated obstacle 3D point cloud of each frame is output.

[0031] Optionally, the correcting the obstacle 3D point cloud of each frame by continuous frame detection includes:

[0032] For each frame of the obstacle 3D point cloud, compare the obstacle point cloud in this frame with that in the adjacent N frames, remove the points that appear less than M times in the adjacent N frames, and then update the obstacle 3D point cloud of this frame; N ≥ 1, 0 < M ≤ N;

[0033] Then, for each frame of the obstacle 3D point cloud, remove the points that appear less than T times in K consecutive frames, and then update the obstacle 3D point cloud of the frame; K>1, 0<T≤K.

[0034] Optionally, the correcting the obstacle 3D point cloud of each frame through continuous frame detection includes:

[0035] Perform clustering on the obstacle 3D point cloud of each frame to extract obstacle clusters;

[0036] Find the obstacle cluster of the previous frame that matches each obstacle cluster in this frame, align each obstacle cluster in this frame to the obstacle cluster of the previous frame that matches it, calculate the optimal rigid body transformation, and consider it a match if the matching error is less than the preset matching threshold;

[0037] Obtain the motion transformation of the matched obstacle cluster in consecutive frames as the motion trajectory;

[0038] If an obstacle cluster can be tracked as a continuous motion trajectory in the time sliding window, the obstacle cluster is retained; otherwise, it is removed, and then the obstacle 3D point cloud of the frame is updated.

[0039] Optionally, before starting the process of correcting the obstacle 3D point cloud of each frame through continuous frame detection, the process further includes:

[0040] The 3D point cloud of obstacles in each frame in the world coordinate system is filtered. The filtering includes: filtering the drivable area, filtering the abnormal points of the point cloud, and downsampling the point cloud.

[0041] The present invention also provides a self - moving device, which is characterized in that:

[0042] It includes at least one camera, as well as a controller and a memory;

[0043] The memory stores an obstacle recognition program, and the controller controls the obstacle recognition program and the camera to implement the steps of the obstacle recognition method of the self - moving device as described above.

[0044] The present invention also provides a computer - readable storage medium, which stores a computer program, and is characterized in that:

[0045] When the computer program is executed by a processor, it implements the steps of the obstacle recognition method of the self - moving device as described above.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] (1) For the obstacle recognition method of the self - moving device of the present invention, first, clear and stable image input is obtained and preliminary correction and filtering processing are performed. Then, the non - grass and non - ground regions in the image are respectively extracted as binary images, and the depth information is extracted as 3D point cloud data. The obstacle 3D point cloud is obtained by screening the 3D point cloud data through the binary image, realizing obstacle recognition. The present invention separately extracts the non - grass binary image and the non - ground binary image to maintain higher semantic clarity and mis - recognition tolerance.

[0048] (2) The present invention directly combines the non - grass binary image and the non - ground binary image to reduce the number of operations, efficiently obtain the obstacle point cloud, and is easy to deploy.

[0049] (3) The present invention first corrects the 3D point cloud data with IMU data, and then uses the obstacle binary image to screen the point cloud to ensure that the obstacle binary image and the 3D point cloud data are aligned in the same coordinate system, thereby achieving spatial consistency and improving the accuracy and reliability of obstacle extraction.

[0050] ((4) The present invention uses the multi - frame IMU cumulative data to extract the stable tilt trend of the self - moving device to correct the error caused by the local shaking of the binocular camera, and further ensures the anti - interference ability and stitching accuracy of the 3D point cloud data.

[0051] (5) The present invention first eliminates the points that appear less than M times in adjacent N frames to achieve local time filtering and eliminate instantaneous false detections, and then eliminates the points that appear less than T times in continuous K frames to achieve global time consistency enhancement and ensure structural stability. The present invention uses local cleaning first and then global stability to improve the overall efficiency and accuracy.

[0052] (6) The present invention realizes multi-frame consistency perception and noise suppression by combining continuous frame matching with distance constraint tracking, and uses a sliding window to construct a time series to improve robustness and trajectory perception. It is suitable for situations where the obstacle is a dynamic deformable body.

[0053] (7) The present invention first filters the 3D point cloud of obstacles in each frame (in the world coordinate system), and then performs continuous frame detection and trajectory correction. The purpose is to clean and regularize the data, improve the stability and efficiency of subsequent multi-frame trajectory matching, and avoid incorrect matching or incorrect trajectory tracking caused by noise interference.

[0054] (8) The present invention extracts point clouds based on image masks, and depth estimation is performed only in key areas to avoid the problem of large-scale texture loss. IMU is used to assist in compensating for geometric inconsistencies caused by texture drift, and point cloud filtering is used to eliminate unreasonable point cloud structures or floating points to reduce misidentification. Finally, continuous frames are used to fill in short-term matching failure areas, thereby significantly improving the accuracy and robustness of obstacle recognition in complex and sparsely textured scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flowchart of steps 1 to 3 in one embodiment of a method for identifying obstacles in a self-moving device of the present invention;

[0056] Figure 2 Flowchart of steps 3.1 to 3.3 in one embodiment of the present invention;

[0057] Figure 3 Flowchart of steps 3.1 to 3.3 in one embodiment of the present invention;

[0058] Figure 4 This is a flowchart of correcting the 3D point cloud of obstacles in each frame through continuous frame detection as described in step 3.2 in one embodiment of the present invention;

[0059] Figure 5 This is a flowchart of correcting the 3D point cloud of obstacles in each frame through continuous frame detection as described in step 3.2 in one embodiment of the present invention;

[0060] Figure 6 This is a flowchart of steps 3.1 to 3.3 in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0062] Reference Figure 1 , a method for identifying obstacles of a self-moving device, comprising the following steps:

[0063] Step 1: Capture an image using at least one camera provided on a mobile device, pre-process the image, and output the pre-processed image;

[0064] It is understood that the camera is a monocular camera or a binocular camera; when there is one camera, it is set at the front end of the self-mobile device; when there are two cameras, they are respectively set at the front end and the rear end of the self-mobile device; when there are more cameras, at least one camera can be set on the side of the self-mobile device; the image refers to the image of the environment surrounding the self-mobile device;

[0065] The above-mentioned preprocessing refers to a series of enhancement, correction and standardization processes performed on the image;

[0066] For example, preprocessing can include image cropping and image normalization;

[0067] This image cropping is used to reduce interference and accelerate inference. Specifically, it involves cropping areas with heights exceeding a threshold, which are irrelevant to the mowing task, while retaining areas with heights less than or equal to the threshold. Alternatively, a segmentation model can be used to automatically identify ground areas and dynamically generate ROIs (Regions of Interest). The segmentation model can be selected from existing models based on needs.

[0068] The above image normalization is used to standardize the input data distribution. Specifically, it includes: updating the pixel value to (pixel value - mean) / variance; it can also partially eliminate the influence of lighting changes by normalizing the R, G, and B components of each pixel;

[0069] In other embodiments, the above-mentioned preprocessing may include: illumination smoothing, boundary enhancement;

[0070] The above-mentioned lighting smoothing specifically includes: in outdoor lighting scenes, removing bright areas or shadows to avoid segmentation errors; in outdoor shadow scenes, using adaptive histogram equalization algorithms (such as CLAHE (Contrast Limited Adaptive Histogram Equalization)) to enhance the local contrast of the image, and then using Gaussian filtering to further smooth the noise;

[0071] The above-mentioned boundary enhancement is used to improve the recognition ability of weak feature obstacles (such as low-contrast, blurred or locally complex texture objects). Specifically, it includes: enhancing the contrast of edge areas by calculating the gradient or second-order derivative of the image;

[0072] In other embodiments, the above preprocessing may include: performing dedistortion and stereo rectification on the images captured by each camera (binocular camera) to ensure that the left image and the right image are aligned during disparity calculation;

[0073] The above-mentioned dedistortion is used to improve the geometric distortion of the image caused by the physical characteristics of the camera lens or installation errors. Specifically, it includes: obtaining the distortion coefficients of the corresponding binocular camera through calibration, using inverse mapping and bilinear interpolation to generate the corrected image; or using undistort() or initUndistortRectifyMap() in OpenCV to directly apply the calibration parameters for correction;

[0074] The above stereo correction is used to eliminate parallax interference and enhance 3D matching accuracy. Specifically, it includes: using stereoRectify() in OpenCV to calculate the correction parameters, and then remapping the image through remap();

[0075] Step 1 specifically involves: capturing an image using at least one camera on the mobile device. The image is saved in a standard format (e.g., .jpg, .png) as "raw_image_XXX.jpg" with the frame number and timestamp recorded (stored in frame_time_map.json) to ensure alignment with subsequent data; performing preprocessing on the image, including but not limited to distortion correction, image enhancement, and normalization; saving the preprocessed image as "processed_image_XXX.png"; and saving metadata such as camera intrinsics, distortion parameters, and image size used in the preprocessing as "camera_intrinsics.yaml."

[0076] Step 2: Based on the image preprocessed in step 1, the corresponding non-grass binary map, non-ground binary map, and 3D point cloud data are output; by independently extracting the non-grass binary map and non-ground binary map, higher semantic clarity and misrecognition tolerance are maintained;

[0077] Understandably, the binary mask (black & white binary mask) is used to classify each pixel in the image;

[0078] The non-grass binary image can be obtained through a semantic segmentation model, such as DeepLabv3+ (Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation), BiSeNet (Bilateral Segmentation Network for Real-time Semantic Segmentation), etc. Specifically, a pre-trained semantic segmentation model is used, and the pre-processed image (one of the left and right images of each frame captured by each binocular camera or each monocular image captured by a monocular camera) is used as input to output the non-grass area, which is then converted into a non-grass binary image.

[0079] In other embodiments, the non-grass binary image can also be obtained based on a color space combined with clustering; the color space is HSV (hue (H), saturation (S), value (V)) or LAB (CIELAB Color Space), and the clustering can be K-means (K-Means Clustering Algorithm).

[0080] The above-mentioned non-ground binary image can be obtained by a semantic segmentation model or a monocular depth estimation network. The monocular depth estimation network can be MiDaS (Multi-scale Interactive Depth-aware Segmentation, multi-scale interactive depth perception segmentation model), Monodepth2 (Learning Monocular Depth Estimation from Stereo Videos), DPT (Depth Prediction Transformer; Specifically, this embodiment uses a pre-trained monocular depth estimation network, including:

[0081] The preprocessed image is used as input and a depth map is output. The depth map is back-projected into a 3D point cloud in the camera coordinate system based on the camera internal parameters. RANSAC (Random Sample Consensus) is then used to fit a plane model in the 3D point cloud, and points whose distance from the fitted plane is less than a corresponding threshold are regarded as ground points, thereby obtaining the ground plane. Points whose vertical distance from the ground plane is less than a corresponding threshold (which can be set to 5 cm) are then determined to be ground points, and mapped back to the above input to generate a non-ground binary map. In other embodiments, geometric thresholds, lightweight neural networks, or slope methods may be used instead of RANSAC.

[0082] In other embodiments, a pre-trained monocular depth estimation network may be used, with a pre-processed monocular image as input, to output a depth map. Each column of the depth map is traversed upwards, and the depth difference between the current pixel and the pixel above it is compared. If the depth difference is greater than a corresponding threshold (which can be set to 0.1m), all pixels above the current pixel in the current column are marked as non-ground areas. After the traversal is completed, a non-ground binary map is output.

[0083] If the camera is a binocular camera, the method for obtaining the 3D point cloud data includes:

[0084] Using a stereo matching algorithm (such as SGBM (Semi-Global Block Matching)), we find the matching pixels of the same object in each left and right image captured by each binocular camera after preprocessing. We calculate their disparity on the horizontal axis of the image and convert the disparity into depth to obtain the coordinates of each pixel and its depth, i.e., the 3D point cloud data in the camera coordinate system.

[0085] If the camera is a monocular camera, the method for obtaining the 3D point cloud data includes:

[0086] Depth estimation is performed on the monocular image using a neural network model to predict the distance from each pixel to the corresponding camera, and then back-projected into 3D point cloud data using the corresponding camera's intrinsic parameters (focal length, principal point). In other embodiments, other existing technologies may also be used;

[0087] Step 2 is as follows: the image preprocessed in step 1 is input into two independent image semantic recognition modules, and the binary images of the non-grass area and the non-ground area are extracted respectively in .png, .npy or .mat formats, where the pixel value is 0 (background) or 1 (target); the model parameters or network ID of each type of semantic network (such as non_grass_net_v2) are also recorded in semantic_config.json for verification; at the same time, the 3D point cloud data corresponding to each frame of the image is obtained from the binocular camera or monocular depth estimation module and saved as raw_pointcloud_XXX.pcd; the 3D point cloud data includes: mapping pairs consisting of image pixels and corresponding depth values, or depth maps generated by the mapping pairs;

[0088] Step 3: Combine the non-grass binary image, the non-ground binary image, and the 3D point cloud data to output the obstacle 3D point cloud;

[0089] Step 3 specifically involves mapping the pixel locations corresponding to obstacles in the non-grass binary image and the non-ground binary image back to 3D point cloud data, retaining only the point coordinates corresponding to these pixels. The obstacle 3D point cloud is output in a structured format that includes the obstacle's 3D bounding box information and type label, and is recorded in local storage or sent to a navigation system or remote server through a communication interface.

[0090] All processing results are saved in the log file obstacle_processing_log.txt, including input frame number, fusion rule, projection success rate, output point statistics, etc.

[0091] It should be noted that the aforementioned autonomous devices are intelligent electronic devices with autonomous mobility capabilities, meeting the definition of "robot" in GB / T 39405-2020. Depending on the application area, they can be personal or household service robots. While the application scenario for these autonomous devices is lawn mowing, they can also be expanded to other tasks, such as agricultural tasks such as orchard picking vehicle inspections and farmland patrol monitoring, urban and park service tasks such as automated cleaning and unmanned delivery, and construction and construction tasks such as site transportation and temporary navigation.

[0092] In some embodiments, in step 1, the image is an image of continuous frames;

[0093] Reference Figure 2 , step 3 includes:

[0094] Step 3.1, align and merge the non-grass binary image and non-ground binary image of each frame to obtain the obstacle binary image of each frame;

[0095] The non-grass area binary map, the non-ground area binary map, and the obstacle binary map are stored as images or matrices in the format of .png, .npy, or .mat; the obstacle binary map and the corresponding frame image are bound to a unified frame number;

[0096] Specifically, for each frame of image, a binary image of the non-grass area (e.g., non_grass_mask_050.png) and a binary image of the non-ground area (e.g., non_ground_mask_050.png) have been extracted. The image format is a single-channel 8-bit grayscale image or a Boolean matrix in .npy format. These two binary images need to be aligned according to the image size and shooting parameters to ensure pixel-level consistency. The alignment parameters such as image size, image center coordinates, and perspective correction matrix are saved as a structured configuration file (e.g., alignment_config_ 050.json); the merge operation uses the logical "AND" rule to multiply the non-grass binary map and the non-ground binary map pixel by pixel to generate the obstacle binary map corresponding to each frame (for example, obstacle_mask_050.png or .npy), which serves as the basis for subsequent point cloud screening; the merge operation and its parameters should be recorded in the merge log file (for example, mask_merge_log_050.txt), and the obstacle binary map and image frame number should be bound in a unified index map (for example, mask_frame_binding.json);

[0097] Step 3.2: Use the obstacle binary map of each frame to extract the obstacle area in the corresponding 3D point cloud data after correction to obtain the obstacle 3D point cloud of each frame;

[0098] The obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the obstacle's 3D bounding box information; the image mask label indicates whether it is an obstacle and can also include a classification label; the timestamp indicates which frame or moment the point comes from;

[0099] Specifically, the obstacle binary map generated in step 3.1 is spatially fused with the 3D point cloud data of the current frame, including: first, establishing a mapping relationship from image pixel coordinates to 3D point cloud coordinates. This mapping relationship is saved as a point cloud index mapping file (such as pixel_to_point_map_050.json or .npy), and its corresponding camera calibration files are camera_intrinsics.yaml and camera_extrinsics.json;

[0100] Based on the locations of pixels with a "1" value (indicating obstacle areas) in the obstacle binary image, the corresponding points in the 3D point cloud data are located. The obstacle point set consisting of these points is extracted from the 3D point cloud data to form the obstacle 3D point cloud for the current frame (for example, obstacle_cloud_050.pcd or .csv). Intermediate information such as the specific mask filtering conditions, number of matching points, and point percentage can be saved as a .json log file (for example, mask_to_point_filter_log.json).

[0101] Step 3.3: Output the 3D point cloud of obstacles in each frame;

[0102] Specifically, the obstacle 3D point cloud extracted in step 3.2 is structured and output according to the frame number and saved as a .pcd, .csv, or .json format file, such as obstacle_cloud_050.pcd).

[0103] This embodiment directly merges the non-grass binary image and the non-ground binary image to reduce the number of operations, efficiently obtains the obstacle point cloud, and is easy to deploy.

[0104] In other embodiments, in order to improve the accuracy and robustness of the obstacle binary map and identify obstacles that "need to be avoided" without accidentally damaging the grass, the obstacle binary map in step 3.1 may also be optimized. The optimization includes:

[0105] For the binary obstacle map, morphological operations are used to remove small spots and fill small holes to remove isolated pixels, small noise blocks, and abnormal areas; then, areas within the preset area range are retained to remove grass edges or ground defects.

[0106] In some embodiments, the camera includes at least one binocular camera; the self-mobile device is provided with an IMU corresponding to each binocular camera; the above correspondence means that the binocular camera and the IMU are rigidly connected; specifically, when there is one binocular camera, it is provided at the front end of the self-mobile device; when there are two binocular cameras, they are provided at the front end and the rear end of the self-mobile device, respectively; the above image includes a left image and a right image captured by each binocular camera; at least one monocular camera may also be provided on the side of the self-mobile device, and the above image also includes a monocular image captured by each monocular camera;

[0107] In step 1, the image is an image of continuous frames;

[0108] Reference Figure 3 , step 3 includes:

[0109] Step 3.1, align and merge the non-grass binary image and non-ground binary image of each frame to obtain the obstacle binary image of each frame;

[0110] Use IMU data to correct the 3D point cloud data of each frame to obtain the corresponding corrected 3D point cloud data to eliminate the cumulative error of visual pose estimation and compensate for dynamic environmental interference;

[0111] The above-mentioned alignment and merging of the non-grass binary image and the non-ground binary image of each frame and the above-mentioned correction of the 3D point cloud data of each frame using the IMU data can be performed simultaneously, or one of them can be performed first;

[0112] It can be understood that the corrected 3D point cloud data refers to the 3D point cloud data obtained by performing posture correction (coordinate system transformation) on the original 3D point cloud data by collecting IMU data that matches the timestamp of the current image frame, combining the posture estimation (including acceleration, angular velocity and posture angle, posture angle including roll, pitch and yaw) in the IMU data, and constructing a 3D space rotation matrix; the corrected 3D point cloud data includes: the original acquisition frame number of each point, the spatial coordinate value after posture correction, and the applied rotation matrix identifier;

[0113] Step 3.2: Use the obstacle binary map of each frame to extract the obstacle area in the corresponding 3D point cloud data after correction, and obtain the obstacle 3D point cloud of each frame; the obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the 3D bounding box information of the obstacle;

[0114] Step 3.3: Output the 3D point cloud of obstacles in each frame.

[0115] In some embodiments, in step 1, the number of the binocular camera is one and is provided on the front end of the self-mobile device; the self-mobile device is provided with an IMU corresponding to the binocular camera;

[0116] The above uses IMU data to correct the 3D point cloud data of each frame to obtain the corresponding corrected 3D point cloud data, including:

[0117] Step A1: Collect IMU data of each IMU within a preset collection time and at a preset sampling frequency. For example, if the preset collection time is 0.5 seconds and the IMU data of the above IMU is collected at a preset sampling frequency of 100 Hz, 50 sets of IMU data can be obtained.

[0118] Each set of IMU data contains: timestamp (absolute timestamp of sampling), accel_x (X-axis linear acceleration), accel_y (Y-axis linear acceleration), accel_z (Z-axis linear acceleration), gyro_x (angular velocity around the X-axis), gyro_y (angular velocity around the Y-axis), gyro_z (angular velocity around the Z-axis);

[0119] Step A2: For each IMU, independently calculate the attitude angle of each set of collected IMU data; each set of IMU data for each IMU can obtain a set of attitude angles, and the representation format of each set of attitude angles includes: timestamp, imu_id (corresponding IMU), roll_deg (roll angle), pitch_deg (pitch angle), yaw_deg (yaw angle);

[0120] Based on the average attitude angle within the preset acquisition time, determine whether the mobile device position corresponding to the IMU (the installation position of the IMU on the mobile device) is currently on a slope. If so, trigger the attitude transformation of the 3D point cloud data of the corresponding frame to obtain the corresponding corrected 3D point cloud data. After completion, execute step 3.2; otherwise, execute step 3.2 directly;

[0121] The above method determines whether the position of the mobile device corresponding to the IMU is currently on a slope based on the average attitude angle within the preset acquisition time, including:

[0122] Calculate the average attitude angle of the IMU within the preset acquisition time. The representation format includes: imu_id, avg_pitch (average pitch), avg_roll (average roll);

[0123] Determine whether the average Pitch or Roll exceeds the set tilt threshold. If so, it means that the position of the mobile device corresponding to the IMU is currently on a slope; otherwise, it is not on a slope.

[0124] It is understandable that the corresponding frame mentioned above refers to the frame with a timestamp closest to the center of the preset acquisition time. This is because the preset sampling frequency of the IMU may be out of sync with the frequency of image acquisition by the binocular camera. To avoid the accumulation of correction errors between the previous and next frames due to posture changes (such as device tilt), it is necessary to find the most recent and representative point cloud frame.

[0125] The above-mentioned attitude transformation includes: using the rotation matrix of the IMU attitude to perform a rotation transformation on the 3D point cloud data of the corresponding frame.

[0126] In other embodiments, the number of the binocular cameras is two, which are respectively arranged at the front end and the rear end of the self-moving device; and the self-moving device is provided with two IMUs corresponding to the two binocular cameras respectively.

[0127] In some embodiments, to avoid problems such as missed detection, false detection, and point cloud jumps that may occur when using only a single frame of obstacle point cloud, before starting step 3.3, the following steps are also included:

[0128] Combined with RTK information, each frame of the obstacle's 3D point cloud is converted to a world coordinate system to obtain a 3D point cloud of the obstacle in the world coordinate system. The RTK information includes the RTK raw positioning data, the RTK corresponding attitude and reference points, and the RTK transformation matrix used for conversion. The 3D point cloud of the obstacle in the world coordinate system includes: the position of each point in the world coordinate system (UTM or ENU coordinates aligned with the geographic coordinate origin), the 3D bounding box information of each obstacle, and the timestamp;

[0129] The obstacle 3D point cloud of each frame is corrected through continuous frame detection, and the updated obstacle 3D point cloud of each frame is output; the updated obstacle 3D point cloud of each frame includes: the position of each point in the world coordinate system (after update), the 3D bounding box information of each obstacle, the timestamp, and the update status description (such as whether it has been corrected by continuous frame fusion, and which frame data is used for fusion, etc.).

[0130] It should be noted that the above-mentioned correction of the obstacle 3D point cloud of each frame through continuous frame detection means: for each frame of the obstacle 3D point cloud, the obstacle 3D point cloud of the frame is compared with the obstacle 3D point cloud of the frame and at least one adjacent frame, and the obstacle 3D point cloud of the frame is corrected through motion compensation and association between continuous frames.

[0131] In some embodiments, the above-mentioned correcting the obstacle 3D point cloud of each frame through continuous frame detection includes:

[0132] For each frame's obstacle 3D point cloud, compare the frame with its N adjacent frames and remove points whose number of appearances in the N adjacent frames is less than M. Specifically, a KD-Tree (K-Dimensional Tree) nearest neighbor search can be used to update the obstacle 3D point cloud of the frame; N ≥ 1, 0 < M ≤ N.

[0133] Specifically, each frame of the obstacle point cloud is annotated with a unique obstacle ID or cluster label to facilitate cross-frame tracking and determination of the number of occurrences. By counting the number of obstacle occurrences in N consecutive frames and the number of global obstacle occurrences in K consecutive frames, points with a number of occurrences less than M in N consecutive frames are eliminated. The obstacle 3D point cloud of the frame is updated and the updated version is recorded. In this embodiment, N can be a value between 3 and 5 to eliminate instantaneous false detections in the local time domain.

[0134] In order to improve efficiency, before correcting the obstacle 3D point cloud of each frame through continuous frame detection, the voxels of the obstacle 3D point cloud of each frame may be clustered and then counted in clusters.

[0135] In other embodiments, reference Figure 4 In order to remove random misidentification or short-term interference, the above-mentioned correction of the obstacle 3D point cloud in each frame through continuous frame detection also includes:

[0136] For each frame of the obstacle 3D point cloud, remove the points that appear less than T times in K consecutive frames, and update the obstacle 3D point cloud of the frame; K>1, 0<T≤K.

[0137] Specifically, each frame of the obstacle point cloud is annotated with a unique obstacle ID or cluster label to facilitate cross-frame tracking and determine the number of occurrences. By counting the number of times an obstacle appears in K consecutive frames, points that appear less than T times in K consecutive frames are eliminated. The obstacle 3D point cloud of the frame is updated and the updated version is recorded. In this embodiment, the value of K can be one of 5 to 10 to retain stable obstacles that appear continuously.

[0138] The above steps are applicable to obstacles whose positions and shapes do not change much in space.

[0139] In some embodiments, if the obstacle is a dynamic deformable body, such as a pedestrian, an animal, a deformable object, etc., refer to Figure 5 , the above-mentioned obstacle 3D point cloud of each frame is corrected through continuous frame detection, including:

[0140] Step B1: perform clustering on the obstacle 3D point cloud of each frame to extract obstacle clusters;

[0141] Specifically, clustering is performed on each frame of the obstacle 3D point cloud (saved in .pcd, .bin, or .npy format) using an algorithm such as Euclidean Clustering. The parameters used in the clustering process (including cluster distance threshold, minimum number of points, maximum number of clusters, etc.) are recorded in a structured file (.json or .txt) as a clustering configuration file. Each obstacle cluster generated after clustering is stored or annotated separately (e.g., each cluster corresponds to a .pcd or .json point set index), associated with the original frame number and timestamp, and recorded in the cluster index file (.json).

[0142] Step B2: Find the obstacle cluster in the previous frame that matches each obstacle cluster in the current frame, align each obstacle cluster in the current frame to the matching obstacle cluster in the previous frame, and calculate the optimal rigid body transformation. If the matching error is less than a preset matching threshold, such as RMSE (Root Mean Square Error) < 0.1, the cluster is considered a match.

[0143] Matching each obstacle cluster of the frame to the corresponding obstacle cluster of the previous frame means that the centroid of each obstacle cluster of the frame matches the obstacle cluster of the previous frame whose centroid distance is less than a preset centroid threshold (such as 0.5m);

[0144] The above optimal rigid body transformation can be expressed as:

[0145]

[0146] Where T is the optimal rigid body transformation, are the ith obstacle cluster of this frame and the ith obstacle cluster of the previous frame, respectively. 2 represents the Euclidean distance squared (L2 norm), Represents finding the rigid body transformation that minimizes the loss function;

[0147] The optimal rigid body transformation is used to calculate the average Euclidean distance error of all corresponding points, i.e., the matching error.

[0148] Specifically, for each obstacle cluster in each frame, the obstacle clusters extracted in the previous frame are searched for those whose centroid distance is less than a preset centroid threshold (e.g., 0.5 m), and an inter-frame matching relationship is established;

[0149] The information of each matching pair, including the current frame cluster ID, the previous frame cluster ID and its optimal rigid body transformation matrix (4×4 matrix, stored in .json or .txt) will be recorded;

[0150] At the same time, the matching error (such as RMSE (Root Mean Square Error)) between each pair of obstacle clusters is calculated and saved in .csv or .json format. If the error is less than the preset threshold (such as RMSE < 0.1, which is also stored in the parameter file), the match is considered successful. The matching status and error value are recorded in the matching record file.

[0151] Step B3: Obtain the motion transformation of the matched obstacle cluster in consecutive frames as the motion trajectory; the above motion transformation refers to the change in the position of the obstacle between consecutive frames, that is, the rotation and translation between two frames;

[0152] Specifically, the matched obstacle clusters in consecutive frames are temporally concatenated to generate the motion trajectory of each obstacle cluster. Each motion trajectory is recorded in a .json file, including the cluster ID, the frame number corresponding to the trajectory, the timestamp sequence, and the 3D position or transformation matrix of each frame (such as the center of mass position, the center of the bounding box, etc.).

[0153] Step B4: If an obstacle cluster can be tracked as a continuous motion trajectory in the time sliding window, the obstacle cluster is retained. Otherwise (jumping, no match, large shape difference) is eliminated, and the 3D point cloud of the obstacle in that frame is updated. The time sliding window refers to a continuous fixed number of frames (such as the last 5 frames) or a time length (such as the last 2 seconds). The continuous motion trajectory refers to: at least L consecutive frames exist, L ≥ 2;

[0154] Specifically, the time sliding window parameters (fixed frame number or duration, such as "last 5 frames" or "2 seconds") are saved in the form of a configuration file (.json). Based on the sliding window tracking results, if an obstacle cluster can be continuously tracked within the sliding window for more than a preset number of frames (e.g., L ≥ 2), the obstacle cluster is retained. Otherwise, if there is a matching interruption, motion jump, or excessive shape difference, the cluster is removed. The reasons for removal (such as no_match, shape_jump, short_lifetime) are recorded in the removal record file (.json).

[0155] The final retained obstacle clusters are reassembled into the updated obstacle 3D point cloud of the current frame and output in .pcd, .csv or .npy format. At the same time, the processing status of the frame and all processing history (clustering → matching → tracking → updating) are recorded in the log file (.log, .json).

[0156] In some embodiments, in order to eliminate interference points and redundant points and enhance the stability of subsequent clustering and matching, before correcting the obstacle 3D point cloud of each frame through continuous frame detection, the following steps may also be performed:

[0157] The obstacle 3D point cloud of each frame in the world coordinate system is filtered. The filtering includes: filtering the drivable area, filtering the point cloud abnormal points, and downsampling the point cloud.

[0158] It can be understood that the above-mentioned drivable area filtering means: defining the drivable area as a cube / polygon boundary, performing spatial filtering on each point of the obstacle 3D point cloud, or directly cropping the obstacle 3D point cloud using a binary map of the non-drivable area; the above-mentioned point cloud abnormal point filtering means: checking the neighborhood density (mean distance) of each point in the obstacle 3D point cloud, eliminating points with excessive deviation from the neighborhood distance mean, or eliminating points with a number of points less than a threshold within a specified radius; the above-mentioned point cloud downsampling means: dividing the point cloud space into grid voxels of a fixed size, and retaining a representative point in each grid voxel.

[0159] Specifically, first, the obstacle 3D point cloud of each frame in the world coordinate system (saved in .pcd, .bin or .npy format) is spatially clipped according to the preset drivable area boundary information (such as road polygons or lane grid maps, stored in .geojson, .json or .txt formats); spatial clipping includes: performing regional constraint judgment on each point, filtering out points that fall outside the non-drivable area, retaining points that fall within the drivable range, and generating the first-stage filtered point cloud data (saved as cloud_after_roi.pcd or .csv). The filtered points (i.e., the excluded point set) will also be saved separately (such as filtered_out_of_roi.pcd) for subsequent misjudgment investigation or evidence comparison;

[0160] Perform outlier removal on the point cloud data after the first stage of filtering, including: using statistical filtering or radius-based neighborhood density removal method to calculate local statistical characteristics (such as mean, variance, number of neighbor points, etc.) for each point, and judge points with low density or excessive deviation as outliers; the filtering parameters used (such as search radius, minimum number of neighbors) are saved in the form of a configuration file (such as outlier_params.json); the removed outlier point set will be recorded separately as removed_outliers.pcd, and the retained points will form a new point cloud data cloud_after_outlier_removal.pcd as the processing result of this stage; the relevant point cloud statistical summary (outlier point ratio, number of removed points, number of retained points, etc.) is written to the filtering statistics file (such as filtering_stats.json or .csv) for engineering visualization;

[0161] Perform spatial downsampling on the new point cloud data (cloud_after_outlier_removal.pcd) to reduce the data size and improve the efficiency of subsequent frame matching and clustering. Save the configuration parameters (such as voxel size and sampling method) as voxel_config.json. The difference in point count and spatial distribution before and after downsampling can be statistically archived in a .csv table (such as point_count_diff.csv).

[0162] The final downsampled point cloud is saved in cloud_after_downsampling.pcd or .npy format for continuous frame obstacle detection and correction.

[0163] This embodiment achieves space constraint, noise suppression, and computational efficiency optimization through a filter chain.

[0164] In one embodiment, a method for identifying obstacles in a mobile device includes the following steps:

[0165] Step 1: Capture continuous frames of images from at least one binocular camera on a mobile device, and preprocess each frame of image;

[0166] Step 2: Obtain the corresponding non-grass binary image, non-ground binary image and 3D point cloud data according to each frame of preprocessed image;

[0167] Step 3: Combine the non-grass binary image, non-ground binary image and 3D point cloud data of each frame to output the obstacle 3D point cloud of each frame;

[0168] Reference Figure 6 , step 3 includes:

[0169] Step 3.1, align and merge the non-grass binary image and non-ground binary image of each frame to obtain the obstacle binary image of each frame;

[0170] Use IMU data to correct the 3D point cloud data of each frame to eliminate the cumulative error of visual pose estimation and compensate for dynamic environmental interference;

[0171] Step 3.2: Use the obstacle binary map of each frame to extract the obstacle area in the corresponding 3D point cloud data after correction to obtain the obstacle 3D point cloud of each frame;

[0172] Combined with RTK information, the 3D point cloud of obstacles in each frame is converted to the world coordinate system;

[0173] Filter the 3D point cloud of obstacles in each frame in the world coordinate system;

[0174] Correct the obstacle 3D point cloud in each frame through continuous frame detection;

[0175] Step 3.3: Output the 3D point cloud of obstacles in each frame.

[0176] The specific methods of the above steps in this embodiment refer to the previous embodiment and will not be repeated here.

[0177] Pure binocular vision is prone to errors on flat, pure-color or sparsely textured surfaces. For example, on dead grass and bare land, the system cannot accurately judge the depth due to the lack of valid matching points. This embodiment performs point cloud extraction based on the image mask, and depth estimation is only performed in key areas to avoid the problem of large-scale texture loss. It also uses IMU to assist in compensating for geometric inconsistencies caused by texture drift, and then uses point cloud filtering to eliminate unreasonable point cloud structures or floating points to reduce misidentification. Finally, continuous frames are used to fill in short-term matching failure areas, thereby significantly improving the accuracy and robustness of obstacle recognition in complex and sparsely textured scenes.

[0178] The present invention also provides a self-moving device, including at least one camera, a controller and a memory, wherein the memory stores an obstacle recognition program, and the controller controls the obstacle recognition program and the camera to implement the steps of the obstacle recognition method of the self-moving device.

[0179] The execution function of the controller corresponds one-to-one to the obstacle identification method for the self-moving device in the above embodiment. For the specific definition of the controller, please refer to the definition of the obstacle identification method for the self-moving device above, which will not be repeated here. The execution process of each sub-module in the above-mentioned controller can be referred to the definition of the obstacle identification method for the self-moving device above, which will not be repeated here. It can be implemented in whole or in part by software, hardware and their combination. Each sub-module can be embedded in or independent of the controller in hardware form, or can be stored in the memory of the controller in software form, so that the controller can call and execute the operations corresponding to each of the above sub-modules.

[0180] In some embodiments, the camera includes at least one binocular camera for capturing continuous frames of images; the self-mobile device is provided with an IMU corresponding to each binocular camera; the correspondence means that the binocular camera and the IMU are rigidly connected.

[0181] In some embodiments, the camera includes two binocular cameras respectively arranged at the front end and the back end of the mobile device, for capturing continuous frame images.

[0182] In some embodiments, the camera includes two binocular cameras respectively arranged at the front and back ends of the mobile device, and at least one monocular camera located on the side, for capturing continuous frame images.

[0183] The present invention also provides a readable storage medium storing computer-readable instructions. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium. The readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors implement the steps of the obstacle identification method for the mobile device in the above-mentioned embodiment.

[0184] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through computer-readable instructions. The above-mentioned computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Claims

1. A method for identifying obstacles in a self-propelled device, characterized in that: The steps include: enabling at least one camera provided on the mobile device to capture an image; Outputting corresponding non-grass binary images, non-ground binary images, and 3D point cloud data based on the image; the 3D point cloud data includes mapping pairs consisting of image pixels and corresponding depth values, or a depth map generated from the mapping pairs; Combining the non-grass binary map, the non-ground binary map, and the 3D point cloud data, a 3D point cloud of the obstacle is output; the 3D point cloud of the obstacle is output in a structured format that includes the three-dimensional bounding box information and type label of the obstacle, and is recorded in a local memory or sent to a navigation system or a remote server through a communication interface.

2. The obstacle recognition method for a self-moving device according to claim 1, characterized in that: The images are images of continuous frames; The step of combining the non-grass binary image, the non-ground binary image, and the 3D point cloud data to output the obstacle 3D point cloud includes: Align and merge the non-grass binary map and the non-ground binary map to obtain the obstacle binary map; The obstacle binary map is used to extract the obstacle area corresponding to the 3D point cloud data to obtain the obstacle 3D point cloud; the obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the three-dimensional bounding box information of the obstacle; Then output the obstacle 3D point cloud.

3. The obstacle recognition method for a self-moving device according to claim 1, characterized in that: The camera includes at least one binocular camera; Each of the binocular cameras corresponds to an IMU, and the IMU is set on the mobile device; The images are images of continuous frames; The step of combining the non-grass binary image, the non-ground binary image, and the 3D point cloud data to output the obstacle 3D point cloud includes: Align and merge the non-grass binary image and non-ground binary image of each frame to obtain the obstacle binary image of each frame; Use IMU data to correct the 3D point cloud data of each frame to obtain the corresponding corrected 3D point cloud data; The obstacle binary map of each frame is used to extract the obstacle area in the corresponding 3D point cloud data after correction, and obtain the obstacle 3D point cloud of each frame; the obstacle 3D point cloud includes: the image mask label corresponding to each point, the timestamp, and the 3D bounding box information of the obstacle; Then output the obstacle 3D point cloud for each frame.

4. The obstacle recognition method for a self-moving device according to claim 3, characterized in that: The method of using IMU data to correct the 3D point cloud data of each frame includes: Collect IMU data from each IMU at a preset sampling frequency within a preset acquisition time. For each IMU, independently calculate the attitude angle of each set of collected IMU data; If the position of the mobile device corresponding to a certain IMU is determined to be on a slope based on the average attitude angle within the preset acquisition time, the attitude transformation of the corresponding frame 3D point cloud data is triggered; The corresponding frame refers to the frame whose timestamp is closest to the center moment of the preset acquisition duration.

5. The obstacle recognition method for a self-moving device according to any one of claims 2 to 4, characterized in that: Before outputting the obstacle 3D point cloud of each frame, the process further includes: Combined with RTK information, each frame of the obstacle 3D point cloud is converted to the world coordinate system to obtain the obstacle 3D point cloud in the world coordinate system. The obstacle 3D point cloud in the world coordinate system includes: the position of each point in the world coordinate system, the timestamp, and the 3D bounding box information of each obstacle; The obstacle 3D point cloud of each frame is corrected through continuous frame detection, and the updated obstacle 3D point cloud of each frame is output.

6. The obstacle recognition method for a self-moving device according to claim 5, characterized in that: The method of correcting the obstacle 3D point cloud of each frame through continuous frame detection includes: For each frame of the obstacle 3D point cloud, compare the obstacle point cloud in this frame with that in the adjacent N frames, remove the points that appear less than M times in the adjacent N frames, and then update the obstacle 3D point cloud in this frame; N ≥ 1, 0 < M ≤ N; Then, for each frame of the obstacle 3D point cloud, remove the points that appear less than T times in K consecutive frames, and then update the obstacle 3D point cloud of the frame; K>1, 0<T≤K.

7. The obstacle recognition method for a self-moving device according to claim 5, characterized in that: The method of correcting the obstacle 3D point cloud of each frame through continuous frame detection includes: Perform clustering on the obstacle 3D point cloud of each frame to extract obstacle clusters; Find the obstacle cluster of the previous frame that matches each obstacle cluster in this frame, align each obstacle cluster in this frame to the obstacle cluster of the previous frame that matches it, calculate the optimal rigid body transformation, and consider it a match if the matching error is less than the preset matching threshold; Obtain the motion transformation of the matched obstacle cluster in consecutive frames as the motion trajectory; If an obstacle cluster can be tracked as a continuous motion trajectory in the time sliding window, the obstacle cluster is retained; otherwise, it is removed, and then the obstacle 3D point cloud of the frame is updated.

8. The obstacle recognition method for a self-moving device according to claim 5, characterized in that: The process of correcting the obstacle 3D point cloud of each frame by continuous frame detection also includes: The 3D point cloud of obstacles in each frame in the world coordinate system is filtered. The filtering includes: filtering the drivable area, filtering the point cloud abnormal points, and downsampling the point cloud.

9. A self-propelled device, characterized in that: Includes at least one camera, a controller and a memory; The memory stores an obstacle recognition program, and the controller controls the obstacle recognition program and the camera to implement the steps of the obstacle recognition method for a mobile device according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the steps of the obstacle recognition method for a mobile device according to any one of claims 1 to 8 are implemented.

Images