Target object recognition method, self-moving device, and control method

Abstract

By acquiring the feature data of multiple dimensions, a mapping relationship of the multi-dimensional feature data is established, and the target object is identified according to the mapping relationship and the feature data of multiple dimensions, thereby avoiding the limitation caused by the inaccurate data due to the single sensor, and effectively identifying the target object through deep fusion of the multi-dimensional data, so that the positioning accuracy is high, the complex lawn environment can be adapted, and the operation requirements of real-time path planning and accurate edge control of the self-moving device can be met.

Description

Technical Field

[0001] This application relates to the field of self-moving devices, and in particular to a target object recognition method, a self-moving device, and a control method thereof. Background Technology With the continuous development of computer and communication technologies, autonomous mobile robots such as lawnmowers, sweeping robots, and outdoor service robots are widely used in various work scenarios, greatly facilitating user operations. In autonomous operation scenarios, accurately perceiving the lawn environment and surrounding obstacles in the work area is a core prerequisite for the efficient operation of autonomous mobile robots.

[0002] Typically, self-propelled robots rely on a single LiDAR sensor or a standalone vision system to perceive their surroundings. However, a standalone LiDAR sensor has limitations in environmental perception, is prone to identification errors, and is difficult to adapt to complex lawn operation scenarios. On the other hand, a standalone vision system is greatly affected by changes in lighting, shadows, and lawn texture, cannot acquire depth information, lacks three-dimensional spatial calibration capabilities, and cannot meet accuracy requirements. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the purpose of this application is to provide a target object recognition method and a self-moving device, in order to solve the problem that existing self-moving robots use a single sensor when perceiving the environment, which cannot meet the progress requirements and results in low work efficiency.

[0004] To address the aforementioned technical problems, this application provides a target object recognition method. This method is applied to a self-moving device equipped with multiple detection devices, including a multi-dimensional imaging detection module and a vision module. The target object recognition method includes: Acquire the first feature data and the second feature data of the target area at the current time. The first feature data includes the three-dimensional data output by the detection module of multi-dimensional imaging of the target area's spatial data. The second feature data includes two-dimensional data output by the vision module from the image data of the target area acquired; Establish a mapping relationship between the first feature data and the second feature data in the target area. The mapping relationship is set based on the coordinate relationship of the same three-dimensional point in the target space in two-dimensional space. Based on the second feature data, a first identification result is obtained. The first identification result is represented as two-dimensional data and is used to indicate a first target object and a second target object in the target area. According to the mapping relationship, the first recognition result is mapped to the feature space of the first feature data, and a second recognition result is output. The second recognition result is represented as three-dimensional data and is used to indicate the first target object and the second target object in the target region. A further improvement is that establishing the mapping relationship between the first feature data and the second feature data in the target object recognition method includes: Obtain the coordinates of the point cloud data in the first feature data in the three-dimensional coordinate system; The point cloud data in the first feature data is projected into the pixel coordinate space of the second feature data using preset calibration parameters. Establish a point-to-point correspondence between the point cloud data in the first feature data and the pixel coordinates in the second feature data.

[0005] A further improvement is as follows: obtaining the first recognition result based on the second feature data includes: The second feature data is preprocessed to obtain preprocessed second feature data. The preprocessing includes at least one of filtering, target region cropping and downsampling. Semantic segmentation is performed on the preprocessed second feature data to extract the target object pixel information from the preprocessed second feature data. The semantic segmentation is used to label the first target object and the second target object. The first recognition result is output based on the pixel information of the target object.

[0006] A further improvement is as follows: the second identification result includes: the first target object is the lawn boundary; the second target object is an obstacle; The first identification result includes: the first target object is at least one of lawn boundary, low grass, and tall grass, and the second target object is at least one of non-passage area and obstacle.

[0007] A further improvement is as follows: the target object recognition method outputs a second recognition result based on the mapping relationship and the first recognition result, including: Based on the mapping relationship, the target pixel information is reverse-mapped to the feature space of the first feature data, and the first feature data corresponding to the target pixel information is extracted to form the second recognition result; The second identification result is the boundary of the lawn.

[0008] A further improvement is that the target object identification method further includes: The first feature data is preprocessed to obtain feature data in the first feature data that represents the second target object, wherein the second target object is an obstacle; Based on the feature data of the obstacles and the preset obstacle filtering algorithm, the first feature data of the effective obstacles are obtained.

[0009] A further improvement is as follows: the preset obstacle removal algorithm includes: The first feature data of the obstacle is preprocessed to filter out data that are characterized as ramps or outliers; Cluster the first feature data after filtering out the data to obtain initial obstacle point cloud data.

[0010] A further improvement is that the target object identification method further includes: The initial obstacle point cloud is projected onto the pixel coordinate space of the second feature data according to the mapping relationship; Determine whether the first identification result of the projection area is the first target object, wherein the first target object is low grass or tall grass; If so, the initial obstacle point cloud data belonging to low grass or high grass in the projection area are removed to obtain valid obstacle point cloud data.

[0011] A further improvement is that the target object identification method further includes: Based on the pixel information of the target area representing the obstacle in the initial environment perception result, the mapping relationship is used to back-map to the feature space of the first feature data, and the point cloud representing the obstacle data in the first feature data is supplemented to the effective obstacle point cloud data, and the final obstacle 3D point cloud set is output.

[0012] A further improvement is that the target object identification method further includes: The accuracy of identifying the first and second targets in the second identification result is higher than that of the first identification result.

[0013] This application also provides a self-moving device, the self-moving device comprising: A detection module for multidimensional imaging, wherein the detection module is used to detect first feature data in the target region; A vision module is used to detect second feature data in a target region; The processor is configured to receive the first feature data and the second feature data, and establish a mapping relationship between the first feature data and the second feature data in the target area, wherein the mapping relationship is set based on the coordinate relationship of the same three-dimensional point in the target space in two-dimensional space. The processor is also configured to receive the first feature data and the second feature data, identify the first target object and the second target object according to the mapping relationship, and control the self-moving device to perform path planning according to the identification result; The detection module and the vision module are both installed in the housing of the self-moving device, and there is a preset external parameter calibration relationship between the detection module and the vision module; The detection module and the vision module are configured to be installed as a whole on the self-moving device.

[0014] This application also provides a self-moving device control method, wherein the self-moving device is configured with multiple detection devices, the detection devices including a multi-dimensional imaging detection module and a vision module; The control method includes: Acquire first feature data and second feature data in the target area. The first feature data includes three-dimensional data output by the detection module of multi-dimensional imaging, which collects spatial data of the target area. The second feature data includes two-dimensional data output by the vision module, which collects image data of the target area. Based on the first feature data and the second feature data, the target information is determined; The target information includes at least one of a first target object and a second target object.

[0015] A further improvement is as follows: Based on the first feature data and the second feature data, determine the target information, including: Based on the first feature data, first target feature data is extracted; and / or, based on the second feature data, second target feature data is extracted; wherein, the first target feature data is target point cloud information, and the second target feature data is target pixel information.

[0016] A further improvement is that the control method further includes: Establish a spatial mapping relationship between the first feature data and the second feature data, and assign the spatial mapping relationship to set the target object point cloud information on the two-dimensional plane corresponding to the second feature data, and / or map the target pixel information to the three-dimensional space corresponding to the first feature data.

[0017] Compared with the prior art, this application has the following advantages: by acquiring feature data in multiple dimensions, establishing a mapping relationship between the multi-dimensional feature data, and identifying the target object based on the mapping relationship and the feature data in multiple dimensions, the limitations caused by inaccurate data due to a single sensor are avoided. By deeply fusing the multi-dimensional data, the target object can be effectively identified with high positioning accuracy. It can adapt to complex lawn environments and meet the operational needs of real-time path planning and precise edge control for mobile devices. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a system provided in an embodiment of this application. Figure 2 This is a flowchart of the environmental perception method provided in the embodiments of this application. Figure 3a A preprocessed image provided for embodiments of this application Figure 3b A semantic segmentation image provided in this application embodiment Figure 4 A schematic diagram of a self-moving device provided in an embodiment of this application. Detailed Implementation In the process of developing this application, the applicant discovered that when a self-moving device operates autonomously in a work area (such as a lawn, courtyard, room, etc.), it needs to perceive the environment of the work area, such as the boundary of the lawn or obstacles in the work area, in order to enable the self-moving device to work efficiently.

[0019] Currently, in the field of environmental perception for self-moving devices, LiDAR and visual sensing have become the core technologies for environmental perception. LiDAR, with its advantage in three-dimensional spatial perception, can acquire three-dimensional point cloud information of the environment, accurately capturing the spatial position of terrain undulations and obstacles, giving it a natural advantage in complex terrain recognition. However, pure LiDAR perception mode has limitations in identifying lawn boundaries, easily leading to inaccurate positioning and failing to accurately identify lawn boundaries.

[0020] Visual sensing can acquire two-dimensional image semantic information of the environment and can intuitively distinguish scene categories such as lawns and obstacles. However, visual sensing is prone to segmentation and misjudgment in complex scenes such as sudden changes in lighting, sparse grass, and undulating terrain. It lacks three-dimensional spatial calibration capabilities, has insufficient progress in boundary space positioning, and is difficult to adapt to complex lawn environments.

[0021] To address the aforementioned issues, this application provides a target object recognition method that effectively solves the problems of low environmental perception accuracy caused by the inaccuracy of single LiDAR solutions and the low recognition accuracy of single vision sensing solutions during environmental perception, which is easily affected by environmental factors.

[0022] To make the technical solutions and advantages of the embodiments of this application clearer, the exemplary embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not an exhaustive list of all embodiments.

[0023] It should be noted that the terms "first," "second," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0024] The terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner, and the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0025] The following is a brief description of the application environment of the environmental perception method for self-moving devices provided in this application embodiment. The environmental perception method is applied to self-moving devices. Self-moving devices can be a series of autonomously moving robots / devices, such as lawnmowers / robots, sweeping robots / devices, and outdoor service robots / devices. Different robots / devices perform different tasks in different work areas; for example, sweeping robots / devices perform cleaning tasks indoors, while lawnmowers / devices perform lawnmowing tasks outdoors.

[0026] Please see Figure 1 , Figure 1 A self-moving device 10 is provided in this application embodiment. The self-moving device 10 includes at least a processor 101 and a sensor 102. The processor 101 is connected to the sensor 102. The processor 101 acquires data collected by the sensor 102. The sensor 102 may include a radar vision integrated sensor, etc.

[0027] A radar-vision integrated sensor is a sensor that integrates a lidar sensor and a vision sensor. It can simultaneously acquire three-dimensional point cloud data and image information. Because it is a radar-vision integrated sensor, it collects point cloud data and image information at the same time, so that the timestamps of the point cloud data and image information are aligned, and the point cloud data and image information are also aligned in space.

[0028] The processor 101 is configured to acquire first feature data and second feature data transmitted by the sensor 102, wherein the first feature data is point cloud data in three-dimensional space and the second feature data is image pixel information in two-dimensional space. The processor 101 is configured to perform initial screening of point cloud data, image processing, identification of lawn boundaries and obstacles, and output corresponding environmental perception results. The environmental perception results include, but are not limited to, the lawn boundary contour features and the location of obstacles within the working area.

[0029] Please see Figure 2 The following embodiment uses the aforementioned self-moving device as a 10-bit execution entity to apply the environmental perception method provided in this application to the aforementioned self-moving device. The environmental perception method provided in this application includes the following steps: In this embodiment, environmental perception is used to identify lawn boundaries.

[0030] Step 201: Establish the mapping relationship between the 3D point cloud data output by the lidar sensor and the 2D image data output by the vision sensor.

[0031] Specifically: S1: Sequentially acquire individual 3D point cloud data to generate a single frame of 3D point cloud data.

[0032] During operation within the work area, the self-moving device acquires single-frame origin point cloud data of the current environment through the LiDAR sensor, while the vision sensor acquires single-frame image data. Assume a single-frame point cloud contains N valid LiDAR 3D point clouds. To facilitate subsequent homogeneous coordinate transformation, the coordinates (xi, yi, zi)T of each point in the LiDAR coordinate system (denoted as coordinate system L) are expanded to the homogeneous form (xi, yi, zi, 1)T, where i = 1, 2, ..., N. All N LiDAR 3D point clouds within a single frame are organized into a 4xN homogeneous coordinate matrix. , The format is as follows:

[0033] Each column represents the three-dimensional coordinates of a lidar point cloud in the lidar coordinate system, and the fourth row is all 1s to support subsequent homogeneous transformations.

[0034] S2: Coordinate system transformation, converting the above single-frame 3D point cloud coordinate data into coordinates in the visual sensor coordinate system.

[0035] Specifically, since the lidar sensor and the vision sensor are installed in different positions, it is necessary to construct an extrinsic transformation matrix to make the coordinate system of the lidar sensor and the coordinate system of the vision sensor the same. The extrinsic matrix describes the relative rotation and relative translation between the lidar sensor and the vision sensor.

[0036] In this embodiment, the extrinsic parameter transformation matrix from the lidar sensor to the vision sensor is converted from the rotation matrix. Translation vector Combining them into a 4×4 homogeneous transformation matrix , The format is as follows:

[0037] This matrix describes the rigid body transformation relationship from the radar coordinate system to the camera coordinate system. Extrinsic parameters can be obtained through joint calibration, such as offline calibration using a checkerboard pattern or corner reflector, and remain fixed during use.

[0038] The coordinate matrix composed of the above-mentioned 3D point clouds of the lidar By directly multiplying with the extrinsic transformation matrix, the coordinates of all LiDAR 3D point clouds in the visual coordinate system (denoted as coordinate system C) can be obtained. , The format is as follows:

[0039] Specifically, The form is:

[0040] Where X, Y, and Z are the three-dimensional coordinates in the visual coordinate system.

[0041] To ensure the physical validity of subsequent projections, points located behind the camera (i.e., points with negative or zero depth values) need to be removed. Specifically, the depth vector Z=[ in the third row is extracted. Z 1, Z 2,…, Z [N], retain only columns that satisfy Zi>0, and set a lower depth threshold, such as Zmin=0.1m, to remove noise points that are too close to the vision sensor. The number of remaining valid points after filtering is denoted as M, M≤N, and updated. And the corresponding depth vector.

[0042] S3: Project the 3D point cloud coordinate data in the visual coordinate system to the pixel coordinates.

[0043] Because visual sensors, such as cameras, have their own physical characteristics, such as focal length, lens distortion, and principal point position, these inherent physical characteristics are referred to as the sensor's intrinsic parameters. To obtain two-dimensional image data more accurately, a projection transformation from the visual coordinate system to the image pixel coordinate system is needed to facilitate computation and processing.

[0044] Specifically: Obtain the camera intrinsic parameter K, which has the following form:

[0045] Where fx and fy are the focal lengths in pixels, and (cx, cy) are the coordinates of the principal point, which is usually located at the center of the image.

[0046] Obtain the coordinates of the obtained 3D point cloud of the LiDAR in the visual coordinate system (denoted as coordinate system C). The first three rows form a new non-homogeneous three-dimensional coordinate matrix. The non-homogeneous 3D coordinate matrix and the intrinsic parameter matrix K are normalized by depth to realize the projection of the 3D coordinates onto the pixel image.

[0047]

[0048] in, is a depth matrix, used for depth normalization, has the following form:

[0049] The above pixel matrix is specifically expanded as follows:

[0050] In the formula denotes element-wise division, and each column outputs a set of pixel coordinates (u, v). Element-wise division refers to performing division operations respectively on elements at corresponding positions in two matrices (or vectors) of the same shape. The calculated pixel coordinates (u, v) are rounded to obtain integer pixel coordinates (uint, vint). If the width of the image is W pixels and the height is H pixels, when 0≤uint<W and 0≤vint<H, it is determined that the pixel coordinates are within the image resolution range, which is regarded as a valid projection point. The valid projection points are retained, and finally the mapping relationship between the 3D point cloud data of the LiDAR sensor and the 2D image data of the vision sensor is established, and the association relationship between each valid 3D point cloud and its corresponding image pixel coordinates is recorded.

[0051] Step 202: Acquire 2D image data from the vision sensor to obtain a first recognition result, where the first recognition result includes a first target object and a second target object.

[0052] In this embodiment, the first target object is at least one of low grass (lawn), lawn boundary, tall grass, and non-passable area.

[0053] S1: Acquire a single-frame RGB image via the vision sensor, and perform preprocessing on it.

[0054] Specifically, Gaussian filtering preprocessing is performed on the single-frame RGB image collected by the vision sensor (e.g., a camera) to suppress sensor noise and texture interference in the environment, so as to improve the feature extraction stability of the subsequent semantic segmentation network.

[0055] The RGB image consists of three independent single-channel layers of R (red), G (green), and B (blue), and each channel is a luminance matrix in grayscale form. First, the original color image is separated into three independent single-channel images of red, green, and blue, which do not interfere with each other.

[0056] Select a Gaussian template with an odd size (such as 3×3, 5×5). The weight of each position in the template is pre-assigned according to the Gaussian normal distribution: the weight of the center pixel is the largest, and the weight decreases sequentially towards the adjacent pixels around, the farther the distance, the smaller the weight, and the sum of all weights is normalized.

[0057] Each of the R, G, and B channels is processed individually: A Gaussian kernel is applied to each pixel within the channel, covering that pixel and its surrounding pixels. This process iterates through all pixels in the entire image, performing single-channel smoothing. After Gaussian weighted filtering of all three channels (red, green, and blue), the three processed single-channel images are re-overlaid and merged according to their original positions and color rules to form a new RGB color image. By applying Gaussian filtering and processing to the acquired single-frame RGB image, noise is reduced, improving the stability of subsequent recognition and detection.

[0058] When self-moving devices, such as autonomous lawnmowers, are operating, the lawn area being processed is mainly located in the lower half of the image, that is, the area close to the ground in the camera's field of view. The upper half of the image usually includes the sky, distant buildings, or trees, which are background elements unrelated to the current operation. Therefore, the RGB image after Gaussian filtering is cropped to retain the lower 2 / 3 of the image height and discard the upper 1 / 3 without losing ground and boundary information. This retained height is related to the camera's mounting angle.

[0059] Furthermore, to meet the real-time requirements of the embedded platform, the cropped image is downsampled. During downsampling, pixels are taken at fixed intervals, or multiple pixels within a region are merged into one, and the overall image length and width are proportionally reduced. In this embodiment, if the original image resolution is 1280×720, the cropped size is 1280×480, and it is downsampled to 320×120. Figure 3a As shown, Figure 3a This is the preprocessed image output after Gaussian filtering and size downsampling. Size downsampling reduces the amount of image data, significantly reducing neural network inference time and improving inference speed.

[0060] S2: Perform segmentation and recognition on the preprocessed image and output the first recognition result.

[0061] Specifically, the preprocessed image from the above steps is input into a preset semantic segmentation neural network. During the training phase of this neural network, an image with manually labeled tags is input into the model. These tags include grass pixels, non-travelable area pixels, etc., which are used for subsequent pixel-level inference by the neural network. After the preprocessed image is input into the preset semantic segmentation neural network, pixel-level inference is performed on the preprocessed image to determine which pixels are grass and which pixels are non-travelable areas. When a pixel is determined to be grass, it is recorded as 1; when a pixel is determined to be non-travelable area, it is recorded as 0. Here, 1 and 0 are the segmentation mask for semantic segmentation, used to label the category of the pixel.

[0062] By performing semantic segmentation on the preprocessed image, all grassy areas and non-trafficible areas in the image can be clearly marked, such as... Figure 3b As shown in Figure 3b, the image is a semantic segmentation image, where the green area represents the lawn and the blue area represents the non-traffic area.

[0063] In the semantic segmentation image described above, pixels with a value of 1 (marked as lawn areas) are divided into different independent regions, each region being a connected component. The pixel area of ​​each connected component is calculated. When the pixel area is less than a preset value, it is identified as a noise region and filtered out. In this embodiment, a pixel area less than 100 pixels is identified as a noise region and removed from the semantic segmentation image. Through this connected component analysis, it is possible to automatically identify which pixels belong to the same continuous lawn area (such as lawn areas without obstacles in the middle) and which pixels are isolated small areas (such as false detections caused by the shadows of leaves), ensuring the connectivity of the main lawn area and improving the reliability of subsequent lawn boundary extraction.

[0064] Through the above connected component analysis, we can identify pixels with a mask value of 1 but at least one neighboring pixel with a mask value of 0. These pixels are recorded as lawn boundary information, and the set of these pixels is the lawn boundary.

[0065] In other implementations, since each pixel in the semantically segmented image has its corresponding segmentation mask value, the pixel positions where the mask value changes (i.e., the pixel positions where the mask value changes from 1 to 0) can be found and recorded as the pixel information of the lawn boundary. The set of these pixels where the mask value changes is the lawn boundary. Based on the segmentation mask value in the semantically segmented image, the corresponding lawn boundary pixel information can be effectively extracted, providing a reliable two-dimensional prior for subsequent point cloud mapping.

[0066] Step 203: Based on the mapping relationship in Step 201, the lawn boundary pixel information obtained in Step 202 is reverse-mapped into three-dimensional point cloud information to obtain the second recognition result.

[0067] The specific steps are as follows: S1: Construct the projection mapping table; Specifically, based on the mapping relationship between the 3D point cloud data and the 2D image data in step 201, the pixel coordinates (uint, vint) corresponding to each 3D point cloud are obtained, and the above mapping relationship is stored in a hash table. The input of the hash table is the pixel coordinate (uint, vint), and the value is the 3D coordinates (x, y, z) of the 3D point cloud in the radar coordinate system L.

[0068] S2: Obtain the corresponding 3D point cloud information based on the pixel information of the lawn boundary, and generate the 3D point cloud information of the lawn boundary. Specifically, for each pixel (ui, vi) in the set of lawn boundary pixels obtained in step 202, the hash table is searched to see if a 3D point cloud is projected onto that pixel or into its neighborhood. For each pixel (ui, vi), a projection point is searched within a preset radius R = 2 pixels. If a match is found, the corresponding 3D point cloud coordinates are extracted and added to the 3D boundary point set P, P = {P1, P2, ..., Pm}, where each point represents a 3D coordinate (x, y, z) in the radar coordinate system L. This 3D boundary point cloud is sparse but accurately distributed. Since each matched 3D point is an actual spatial point directly measured by the lidar, its depth and azimuth accuracy are high and it is not affected by monocular vision depth estimation errors. Therefore, the spatial position of the boundary points is accurate and reliable.

[0069] S3: Control the movement of the self-moving device based on the three-dimensional point cloud information of the lawn boundary.

[0070] Specifically, the self-moving device performs path planning or positioning based on the three-dimensional point cloud information of the lawn boundary.

[0071] The above-mentioned method of environmental perception using radar and vision integrated sensors, especially the method of identifying lawn boundaries, deeply integrates visual semantic information with lidar three-dimensional perception, which can effectively identify lawn boundaries with high boundary spatial positioning accuracy, adapt to complex lawn environments, and meet the operational needs of real-time path planning and precise edge control for mobile devices.

[0072] In another embodiment of this application, environmental perception is the identification of obstacles in the work area environment.

[0073] The specific steps are as follows: Step 301: Establish the mapping relationship between the 3D point cloud data output by the lidar sensor and the 2D image data output by the vision sensor.

[0074] Specifically: S1: Sequentially acquire individual 3D point cloud data to generate a single frame of 3D point cloud data.

[0075] During operation within the work area, the self-moving device acquires single-frame origin point cloud data of the current environment through the LiDAR sensor, while the vision sensor acquires single-frame image data. Assume a single-frame point cloud contains N valid LiDAR 3D point clouds. To facilitate subsequent homogeneous coordinate transformation, the coordinates (xi, yi, zi)T of each point in the LiDAR coordinate system (denoted as coordinate system L) are expanded to the homogeneous form (xi, yi, zi, 1)T, where i = 1, 2, ..., N. All N LiDAR 3D point clouds within a single frame are organized into a 4xN homogeneous coordinate matrix. , The format is as follows:

[0076] Each column represents the three-dimensional coordinates of a lidar point cloud in the lidar coordinate system, and the fourth row is all 1s to support subsequent homogeneous transformations.

[0077] S2: Coordinate system transformation, converting the above single-frame 3D point cloud coordinate data into coordinates in the visual sensor coordinate system.

[0078] Specifically, since the lidar sensor and the vision sensor are installed in different positions, it is necessary to construct an extrinsic transformation matrix to make the coordinate system of the lidar sensor and the coordinate system of the vision sensor the same. The extrinsic matrix describes the relative rotation and relative translation between the lidar sensor and the vision sensor.

[0079] In this embodiment, the extrinsic parameter transformation matrix from the lidar sensor to the vision sensor is converted from the rotation matrix. Translation vector Combining them into a 4×4 homogeneous transformation matrix , The format is as follows:

[0080] This matrix describes the rigid body transformation relationship from the radar coordinate system to the camera coordinate system. Extrinsic parameters can be obtained through joint calibration, such as offline calibration using a checkerboard pattern or corner reflector, and remain fixed during use.

[0081] The coordinate matrix composed of the above-mentioned 3D point clouds of the lidar By directly multiplying with the extrinsic transformation matrix, the coordinates of all LiDAR 3D point clouds in the visual coordinate system (denoted as coordinate system C) can be obtained. , The format is as follows:

[0082] Specifically, The form is:

[0083] Where X, Y, and Z are the three-dimensional coordinates in the visual coordinate system.

[0084] To ensure the physical validity of subsequent projections, points located behind the camera (i.e., points with negative or zero depth values) need to be removed. Specifically, the depth vector Z=[ in the third row is extracted. Z 1, Z 2,…, Z[N], retain only columns that satisfy Zi>0, and set a lower depth threshold, such as Zmin=0.1m, to remove noise points that are too close to the vision sensor. The number of remaining valid points after filtering is denoted as M, M≤N, and updated. And the corresponding depth vector.

[0085] S3: Project the 3D point cloud coordinate data in the visual coordinate system to the pixel coordinates.

[0086] Because visual sensors, such as cameras, have their own physical characteristics, such as focal length, lens distortion, and principal point position, these inherent physical characteristics are referred to as the sensor's intrinsic parameters. To obtain two-dimensional image data more accurately, a projection transformation from the visual coordinate system to the image pixel coordinate system is needed to facilitate computation and processing.

[0087] Specifically: Obtain the camera intrinsic parameter K, which has the following form:

[0088] Where fx and fy are the focal lengths in pixels, and (cx, cy) are the coordinates of the principal point, which is usually located at the center of the image.

[0089] Obtain the coordinates of the obtained 3D point cloud of the LiDAR in the visual coordinate system (denoted as coordinate system C). The first three rows form a new non-homogeneous three-dimensional coordinate matrix. The non-homogeneous 3D coordinate matrix and the intrinsic parameter matrix K are normalized by depth to realize the projection of the 3D coordinates onto the pixel image.

[0090]

[0091] in, This is the depth matrix, used for depth normalization. The format is as follows:

[0092] The above pixel matrix The details are as follows:

[0093] In the formula represents element-wise division, and each column outputs a set of pixel coordinates (u, v). Element-wise division refers to performing division operations on elements at corresponding positions in two matrices (or vectors) of the same shape. The calculated pixel coordinates (u, v) are rounded to obtain integer pixel coordinates (uint, vint). If the width of the image is W pixels and the height is H pixels, when 0 ≤ uint < W and 0 ≤ vint < H, it is determined that the pixel coordinate is within the image resolution range, which is regarded as a valid projection point. The valid projection points are retained, and finally the mapping relationship between the 3D point cloud data of the LiDAR sensor and the 2D image data of the vision sensor is established, and the association relationship between each valid 3D point cloud and its corresponding image pixel coordinate is recorded.

[0094] Step S302: preprocessing the above 3D point cloud of the LiDAR; Specifically, the distance between the 3D point cloud and the self-moving device is calculated. When the distance is greater than a first preset distance or less than a second preset distance, the 3D point cloud is determined as a noise point, an isolated point, or redundant information irrelevant to the target scene, and this type of 3D point cloud is filtered out from the above 3D point cloud. In this embodiment, the first preset distance is 2m, and the second preset distance is 0.1m, that is, point clouds that are too close to the self-moving device and too far from the self-moving device are filtered out.

[0095] At the same time, downsampling is performed on the filtered 3D point cloud data, so that only one representative point cloud is retained in each 3D grid of the point cloud space, the number of point clouds is reduced, and the subsequent processing speed is improved.

[0096] Step S303: acquiring first feature data of a valid obstacle, that is, point cloud data of the valid obstacle, according to the preprocessed 3D point cloud; The specific steps are as follows: S1: filtering out non-obstacle point clouds according to the processed 3D point cloud to obtain initial obstacle point clouds; The preprocessed 3D point cloud is projected onto the two-dimensional plane XOY, for example, the ground, and a two-dimensional grid size G is set. In this embodiment, the two-dimensional grid size is 0.1m*0.1m. The lowest Z coordinate, that is, the lowest height, of all points falling into each grid is recorded within each grid.

[0097] A preset rectangular area in front of the self-moving area is defined, for example, a rectangular area with a length of 1m and a width of 0.8m. If the lowest Z coordinate of the grid in the preset rectangular area is greater than 0, it is forcibly corrected to a preset ground height threshold, for example, 0.02m, which represents the surface height of the lawn. This method can correct the ground missing caused by the self-occlusion of the self-moving device.

[0098] To facilitate all grids, the lowest Z-coordinate of each grid is updated to the maximum value between the lowest Z-coordinate and the ground height threshold, ensuring that the ground reference of all grids is at least not lower than the preset ground height, thus avoiding the influence of negative elevation noise.

[0099] Traverse all preprocessed 3D point clouds and obtain their elevation Z. Find the grid corresponding to each 3D point cloud and obtain the ground reference and obstacle height threshold of the grid. If the elevation Z of the 3D point cloud is greater than or equal to the sum of the ground reference and obstacle height threshold of the grid, then the 3D point cloud is determined to be an obstacle point cloud. This type of 3D point cloud is a protrusion at a significant height above the ground. In this embodiment, the obstacle height threshold is 0.05m.

[0100] For the remaining 3D point clouds in the processed 3D point cloud that were not identified as obstacles, they may belong to slopes or flat ground. At a given point cloud location, the Euclidean distance between adjacent point clouds is calculated. If the distance exceeds a threshold (e.g., 0.2m), or the change in direction of motion exceeds an angle threshold (e.g., 30°), the point cloud is segmented into different continuous point cloud segments. For each continuous point cloud segment, its point cloud coordinates are transformed into 3D coordinates, and the slope angle is calculated. If the slope angle is greater than a preset angle threshold (e.g., 15°), the segment is determined to be a slope, not undetected sparse noise, and all point clouds within that segment are identified as obstacle point clouds. These obstacle point clouds are then used as the initial obstacle point cloud and output.

[0101] S2: Preprocess the initial obstacle point cloud to obtain a preprocessed obstacle point cloud.

[0102] Specifically, the search radius of the neighborhood is set to r, r=0.05m. The entire point cloud space is divided into a three-dimensional cubic grid with a side length of r. Each point cloud in the initial obstacle point cloud obtained in S1 above is inserted into this grid to establish a mapping from the grid to the point list, thereby reducing the complexity of subsequent neighborhood search.

[0103] Because point clouds have inherent distance information: the closer to the lidar, the more point clouds there are; the farther away from the lidar, the fewer point clouds there are. For example, the number of point clouds 0.1m away from the lidar is 100, while the number of point clouds 0.2m away from the lidar is 25. Therefore, the minimum number of effective neighboring points is dynamically calculated for each point cloud to avoid the accidental deletion of sparse and effective point clouds at a distance.

[0104] For each obstacle point cloud, the system iterates through it and quickly obtains all neighborhood points within a radius of r=0.05m around the point cloud using a 3D mesh index, then counts the number of actual valid neighborhood points. The count of actual valid neighborhood points is compared with the calculated minimum number of valid neighborhood points. If the number of actual valid neighborhood points is less than the minimum number, the point is considered a sparse outlier and is removed; otherwise, if the number of actual valid neighborhood points is greater than or equal to the minimum number, the point is considered a dense point and is retained. The preprocessed obstacle point cloud is then output.

[0105] S3: Obtain the final obstacle point cloud based on the preprocessed obstacle point cloud; Specifically, Euclidean clustering is performed on the preprocessed obstacle point cloud in step S2 to form independent obstacle clusters.

[0106] Set a proximity threshold D, for example, 0.5m. Calculate the Euclidean distance from the preprocessed obstacle point cloud to the origin of the sensor coordinate system and calculate its average distance d. If the average distance d ≤ D, it is judged as a proximity point cloud. When clustering proximity point clouds, set a small cluster radius R, R = 0.01m, and a small minimum number of points N1 in the cluster, for example, N1 = 5. Retain all preprocessed obstacle point clouds that meet the above conditions, retaining low obstacles with small surface areas, such as stones and small wooden stakes.

[0107] For other point clouds, calculate the Euclidean distance from the preprocessed obstacle point cloud to the origin of the sensor coordinate system, and calculate its average distance d. Based on the point cloud's own distance information, dynamically calculate the minimum number of points N2 for clustering, which enables effective clustering of sparse obstacles at long distances.

[0108] For each cluster, its height, width, and number of points are calculated. If its height is less than a preset value and the number of points is greater than an adaptive threshold, it is determined to be a valid obstacle cluster. The first feature data of the valid obstacle, i.e., the point cloud data of the valid obstacle, is output through coordinate optimization and downsampling.

[0109] Step S304: Acquire two-dimensional image data from the vision sensor to obtain a first recognition result, the first recognition result including a first target object and a second target object.

[0110] In this embodiment, the first target object is at least one of low grass (lawn), lawn boundary, tall grass, and non-traffic area.

[0111] Specifically: S1: Acquire a single frame of RGB image through a visual sensor and preprocess it.

[0112] Specifically, Gaussian filtering is performed on single-frame RGB images captured by visual sensors (such as cameras) to neutralize sensor noise and texture interference in the environment, thereby improving the feature extraction stability of subsequent semantic segmentation networks.

[0113] An RGB image consists of three independent single-channel layers: R (red), G (green), and B (blue). Each channel is a grayscale brightness matrix. First, the original color image is separated into three independent single-channel images: red, green, and blue, without interference between them.

[0114] Select a Gaussian template of odd size (such as 3×3 or 5×5). Each position in the template is pre-assigned a weight according to a Gaussian normal distribution: the center pixel has the largest weight, which decreases sequentially to the surrounding adjacent pixels. The weight decreases as the distance increases, and the overall weight is normalized.

[0115] Each of the R, G, and B channels is processed individually: A Gaussian kernel is applied to each pixel within the channel, covering that pixel and its surrounding pixels. This process iterates through all pixels in the entire image, performing single-channel smoothing. After Gaussian weighted filtering of all three channels (red, green, and blue), the three processed single-channel images are re-overlaid and merged according to their original positions and color rules to form a new RGB color image. By applying Gaussian filtering and processing to the acquired single-frame RGB image, noise is reduced, improving the stability of subsequent recognition and detection.

[0116] When self-moving devices, such as autonomous lawnmowers, are operating, the lawn area being processed is mainly located in the lower half of the image, that is, the area close to the ground in the camera's field of view. The upper half of the image usually includes the sky, distant buildings, or trees, which are background elements unrelated to the current operation. Therefore, the RGB image after Gaussian filtering is cropped to retain the lower 2 / 3 of the image height and discard the upper 1 / 3 without losing ground and boundary information. This retained height is related to the camera's mounting angle.

[0117] Furthermore, to meet the real-time requirements of the embedded platform, the cropped image is downsampled. During downsampling, pixels are taken at fixed intervals, or multiple pixels within a region are merged into one, and the overall image length and width are proportionally reduced. In this embodiment, if the original image resolution is 1280×720, the cropped size is 1280×480, and it is downsampled to 320×120. Figure 3a As shown, Figure 3a This is the preprocessed image output after Gaussian filtering and size downsampling. Size downsampling reduces the amount of image data, significantly reducing neural network inference time and improving inference speed.

[0118] S2: Perform segmentation and recognition on the preprocessed image and output the recognition results.

[0119] Specifically, the preprocessed image from the above steps is input into a preset semantic segmentation neural network. During the training phase of this neural network, an image with manually labeled tags is input into the model. These tags include grass pixels, non-travelable area pixels, etc., which are used for subsequent pixel-level inference by the neural network. After the preprocessed image is input into the preset semantic segmentation neural network, pixel-level inference is performed on the preprocessed image to determine which pixels are grass and which pixels are non-travelable areas. When a pixel is determined to be grass, it is recorded as 1; when it is determined to be a non-travelable area, it is recorded as 0; when it is determined to be an obstacle, it is recorded as 2; and when it is determined to be tall grass, it is recorded as 3. Here, 0, 1, 2, and 3 are the segmentation masks for this semantic segmentation, used to label the category of the pixel.

[0120] By performing semantic segmentation on the preprocessed image, all grassy areas, non-trafficible areas, and obstacle areas in the image can be clearly labeled. Connectivity analysis is then used to filter out false positive noise areas with small surface areas, ultimately resulting in accurate extraction of segmented pixel information, providing a reliable two-dimensional prior for subsequent point cloud mapping.

[0121] Step S305: Based on the mapping relationship in step S301, the obstacle pixel information in step S304 is reverse-mapped into three-dimensional point cloud information.

[0122] Specifically: S1: Filter out non-obstacle point cloud data from the valid obstacle point cloud data; Based on the relationship between the 3D point cloud and the 2D pixels in step S301, the final obstacle point cloud is projected onto the image obtained in step S304 to obtain the corresponding pixel set. Based on the segmentation mask value of each pixel, it is determined that the pixel is equal to 1 or 3, that is, the pixel belongs to the lawn or tall grass category. The number of lawn or tall grass pixels is counted. If the proportion of the number of pixels of the obstacle cluster to the total number of pixels of the obstacle cluster to which the pixel belongs exceeds a preset threshold, for example, more than 60% of the pixels of the obstacle cluster are marked as lawn or tall grass, then the obstacle cluster is determined to be tall grass and not an obstacle, and is removed from the obstacle point cloud.

[0123] Based on the mapping relationship between the 3D point cloud data and the 2D image data in step 301, the pixel coordinates (uint, vint) corresponding to each 3D point cloud are obtained. The above mapping relationship is stored in a hash table. The input of the hash table is the pixel coordinate (uint, vint), and the value is the 3D coordinates (x, y, z) of the 3D point cloud in the radar coordinate system L.

[0124] For each obstacle pixel (all pixels with a segmentation mask of 2) obtained in step S304, its corresponding LiDAR point cloud index is obtained through reverse mapping. All these LiDAR points are collected and added to the final obstacle point cloud set.

[0125] S3: Control the movement of the self-moving device based on the point cloud information output by the final obstacle.

[0126] Specifically, the self-moving device controls obstacle avoidance based on the point cloud information output by the final obstacle.

[0127] By using the aforementioned environmental perception method for self-moving devices, obstacles within the working area are perceived, achieving deep fusion of visual semantic information and LiDAR three-dimensional perception. This effectively solves problems such as false detection of tall grass, false detection of slopes, and insufficient real-time performance, accurately detecting obstacles and improving the working efficiency of self-moving devices.

[0128] The target object recognition method provided in this application establishes a mapping relationship between radar 3D point clouds and 2D pixels, and identifies the working area environment of the self-moving device through dual verification of radar 3D point clouds and 2D pixels, thereby improving the accuracy of identifying the working area environment. At the same time, it takes into account depth and semantic information, significantly improving the working efficiency and intelligence level of the self-moving device.

[0129] This application also provides a self-moving device, which includes a multi-dimensional imaging detection module and a vision module. In this embodiment, the multi-dimensional imaging detection module is a lidar sensor, and the vision module is a camera. In this application, the lidar sensor and the camera are integrated together and arranged at the front end of the self-moving device.

[0130] The lidar sensor is used to detect the first feature data in the working area, namely the point cloud data of the target object in three-dimensional space, while the camera is used to detect the second feature data in the working area, namely the image pixel data of the target object in two-dimensional space.

[0131] The lidar sensor is integrated with the camera and connected to a processor. The processor acquires point cloud data and pixel data for mapping and processing, and is used to identify the first and second targets in the working area. Targets include at least one of low grass (lawn), tall grass, lawn boundaries, obstacles, and non-travelable areas. Based on the identification results, the self-moving device is controlled to perform path planning, obstacle avoidance, and other tasks.

[0132] Both the lidar sensor and the camera are mounted on the housing of the self-moving device, either on the housing or inside it. There is a preset extrinsic calibration relationship between the lidar sensor and the camera. The preset extrinsic calibration relationship is as follows: the lidar sensor acquires the point cloud data of the target object in three-dimensional space, that is, it acquires the point cloud coordinates in the lidar coordinate system. The point cloud data is then formed into a point cloud matrix and transformed into image pixel data in the camera coordinate system, that is, pixel coordinates, through the preset extrinsic parameter matrix.

[0133] LiDAR sensors and cameras can be installed as a single unit on a self-moving device.

[0134] This application also provides a control method for a self-moving device, applied to the self-moving device described above, which acquires point cloud data and pixel data in the working area, and can identify target objects based on the point cloud data and pixel data. Target objects include any one of low grass (lawn), tall grass, lawn boundaries, obstacles, and non-travelable areas.

[0135] Specifically, during the movement of the mobile device, the LiDAR sensor acquires point cloud data of the target object in each frame, and the camera acquires pixel data of the target object in each frame. According to the mapping relationship mentioned in the above embodiments, the point cloud data of the target object is mapped onto a two-dimensional plane corresponding to the pixel data, or the pixel data of the target object is mapped onto a three-dimensional space corresponding to the point cloud data. For example, when the target object identified based on the point cloud data is an obstacle, the point cloud data of the obstacle is projected onto the two-dimensional plane where the pixel data is located according to a preset mapping relationship. In this two-dimensional plane, if the identification result corresponding to the pixel data is high grass / low grass, then the point cloud data is removed. This achieves dual verification of point cloud data and pixel data, improving recognition accuracy.

[0136] Please see Figure 4 This application embodiment also provides a self-moving device 40, which includes a memory 401 and a processor 402.

[0137] The memory 401 can be a Flash chip, a read-only memory (ROM), a disk, an optical disk, a USB flash drive, or a portable hard drive, etc.

[0138] The processor 402 can be a microcontroller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP), etc.

[0139] The memory 401 stores a computer program; the processor 402 executes the computer program to implement the environmental perception method of the self-moving device 40 described above. Specific details of the cliff detection and control methods of the self-moving robot 40 are given in the foregoing embodiments and will not be repeated here.

[0140] Finally, it should be noted that although the operations are described in a specific order in the accompanying drawings, this should not be construed as requiring the operations to be performed in the specific order shown or in a sequential order, or to perform all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

[0141] The above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. A target object recognition method, wherein the target object recognition method is applied to a self-moving device, characterized in that, The self-moving device is equipped with multiple detection devices, including a multi-dimensional imaging detection module and a vision module. The target object recognition method includes: Acquire the first feature data and the second feature data of the target area at the current time. The first feature data includes the three-dimensional data output by the detection module of multi-dimensional imaging of the target area's spatial data. The second feature data includes two-dimensional data output by the vision module from the image data of the target area acquired; Establish a mapping relationship between the first feature data and the second feature data in the target area. The mapping relationship is set based on the coordinate relationship of the same three-dimensional point in the target space in two-dimensional space. Based on the second feature data, a first identification result is obtained. The first identification result is represented as two-dimensional data and is used to indicate a first target object and a second target object in the target area. According to the mapping relationship, the first recognition result is mapped to the feature space of the first feature data, and a second recognition result is output. The second recognition result is represented as three-dimensional data and is used to indicate the first target object and the second target object in the target area.

2. The target object identification method according to claim 1, characterized in that, The target object recognition method includes establishing the mapping relationship between the first feature data and the second feature data, which includes: Obtain the coordinates of the point cloud data in the first feature data in the three-dimensional coordinate system; The point cloud data in the first feature data is projected into the pixel coordinate space of the second feature data using preset calibration parameters. Establish a point-to-point correspondence between the point cloud data in the first feature data and the pixel coordinates in the second feature data.

3. The target object identification method according to claim 1, characterized in that, The step of obtaining the first recognition result based on the second feature data includes: The second feature data is preprocessed to obtain preprocessed second feature data. The preprocessing includes at least one of filtering, target region cropping and downsampling. Semantic segmentation is performed on the preprocessed second feature data to extract the target object pixel information from the preprocessed second feature data. The semantic segmentation is used to label the first target object and the second target object. The first recognition result is output based on the pixel information of the target object.

4. The target object identification method according to claim 3, characterized in that, The target object recognition method, in which a second recognition result is output based on the mapping relationship and the first recognition result, includes: Based on the mapping relationship, the target pixel information is reverse-mapped to the feature space of the first feature data, and the first feature data corresponding to the target pixel information is extracted to form the second recognition result; The second identification result is the boundary of the lawn.

5. The target object identification method according to claim 1, characterized in that, The second identification result includes: the first target object is the lawn boundary; the second target object is an obstacle; The first identification result includes: the first target object is at least one of lawn boundary, low grass, and tall grass, and the second target object is at least one of non-passage area and obstacle.

6. The target object identification method according to claim 1, characterized in that, The target object identification method further includes: The first feature data is preprocessed to obtain feature data in the first feature data that represents the second target object, wherein the second target object is an obstacle; Based on the feature data of the obstacles and the preset obstacle filtering algorithm, the first feature data of the effective obstacles are obtained.

7. The target object identification method according to claim 6, characterized in that, The preset obstacle removal algorithm includes: The first feature data of the obstacle is preprocessed to filter out data that are characterized as ramps or outliers; Cluster the first feature data after filtering out the data to obtain initial obstacle point cloud data.

8. The target object identification method according to claim 7, characterized in that, The target object identification method further includes: The initial obstacle point cloud is projected onto the pixel coordinate space of the second feature data according to the mapping relationship; Determine whether the first identification result of the projection area is the first target object, wherein the first target object is low grass or tall grass; If so, the initial obstacle point cloud data belonging to low grass or high grass in the projection area are removed to obtain valid obstacle point cloud data.

9. The target object identification method according to claim 8, characterized in that, The target object identification method further includes: Based on the pixel information of the target area representing the obstacle in the initial environment perception result, the mapping relationship is used to back-map to the feature space of the first feature data, and the point cloud representing the obstacle data in the first feature data is supplemented to the effective obstacle point cloud data, and the final obstacle 3D point cloud set is output.

10. The target recognition method according to claim 1, characterized in that, The target object identification method further includes: The accuracy of identifying the first and second targets in the second identification result is higher than that of the first identification result.

11. A self-moving device, characterized in that, The self-moving device includes: A detection module for multidimensional imaging, wherein the detection module is used to detect first feature data in the target region; A vision module is used to detect second feature data in a target region; The processor is configured to receive the first feature data and the second feature data, and establish a mapping relationship between the first feature data and the second feature data in the target area, wherein the mapping relationship is set based on the coordinate relationship of the same three-dimensional point in the target space in two-dimensional space. The processor is also configured to receive the first feature data and the second feature data, identify the first target object and the second target object according to the mapping relationship, and control the self-moving device to perform path planning according to the identification result; The detection module and the vision module are both installed in the housing of the self-moving device, and there is a preset external parameter calibration relationship between the detection module and the vision module; The detection module and the vision module are configured to be installed as a whole on the self-moving device.

12. An automatic equipment control method, characterized in that, The self-moving device is equipped with multiple detection devices, including a multi-dimensional imaging detection module and a vision module; The control method includes: Acquire first feature data and second feature data in the target area. The first feature data includes three-dimensional data output by the detection module of multi-dimensional imaging, which collects spatial data of the target area. The second feature data includes two-dimensional data output by the vision module, which collects image data of the target area. Determine target information based on the first feature data and the second feature data. The target information includes at least one of a first target object and a second target object.

13. The self-moving device control method according to claim 11, characterized in that, Based on the first feature data and the second feature data, target information is determined, including: Based on the first feature data, first target feature data is extracted; and / or, based on the second feature data, second target feature data is extracted; wherein, the first target feature data is target point cloud information, and the second target feature data is target pixel information.

14. The self-moving device control method according to claim 13, characterized in that, The control method further includes: Establish a spatial mapping relationship between the first feature data and the second feature data, and assign the spatial mapping relationship to map the target object point cloud information onto the two-dimensional plane corresponding to the second feature data, and / or map the target pixel information onto the three-dimensional space corresponding to the first feature data.

Images