A target identification method and device, electronic equipment and storage medium

Abstract

The application provides a target identification method and device, electronic equipment and storage medium. The method comprises the following steps: screening seed points according to the first distance between the second pixel points obtained by projecting the original point cloud and the center point of the target mask region and the depth value of the original point corresponding to the second pixel points; determining an adaptive search radius according to the second distance between the seed points and the original points corresponding to each second pixel point; performing adaptive region growing based on the adaptive seed points and the search radius to search the original points on the target; and determining a three-dimensional perception result of the target based on all the searched original points. The application can obtain adaptive seed points and search radius based on the characteristics of different targets to form an adaptive region growing algorithm. Then, the target can be identified to meet the search requirements of different targets and improve the target identification accuracy and speed.

Description

Technical Field

[0001] This invention relates to the field of environmental perception for quadruped robots, and more specifically, to a target recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Environmental perception is crucial for quadruped robots to navigate autonomously and identify targets in complex environments. It enables robots to effectively avoid obstacles and accurately identify targets of interest. Researching multi-modal fusion perception techniques using 2D perception results and raw point clouds allows robots to accurately and efficiently acquire 3D perception information of targets.

[0003] However, 2D perception results are generated based on 2D images, while point clouds consist of multiple unordered 3D points, creating an inherent difficulty in matching the two. When using 2D perception results to filter 3D point clouds, noisy point clouds are inevitably introduced. Therefore, the filtered point clouds must be clustered to remove noisy point clouds and retain the pure target point cloud. Traditional clustering methods include search clustering methods based on local similarity features and density-based clustering methods.

[0004] Density-based clustering methods, such as DBSCAN, distinguish between core points, boundary points, and noise points by setting a search radius and a minimum number of neighborhood points, and expand the clustering region by density. Starting from the core point, all density-reachable points are grouped into one class until no more points are added. This method can automatically detect the number of clusters and identify noise points. However, when the scene contains only one target cluster and multiple noise clusters, this method exhibits information redundancy and is time-consuming, reducing the speed of target recognition.

[0005] Region growing is a search-based clustering method based on local similarity features. It sets a seed point and a neighborhood range, starting from a seed point and progressively adding neighboring similar points to the current cluster until no more similar points are found. This method can effectively handle non-convex clusters and ignore noise. However, its limitation lies in the need to manually design fixed seed points and fixed radii. When dealing with point clouds containing multiple targets of varying distances and sparsity, fixed seed points and fixed radii often fail to simultaneously meet the search requirements of different targets, reducing target recognition accuracy. Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a target recognition method, apparatus, electronic device, and storage medium that can obtain adaptive seed points and search radii based on the characteristics of different targets, forming an adaptive region growing algorithm, and then identifying the target. Compared with the traditional target recognition method based on fixed seed points and fixed radii, this application can meet the search requirements of different targets when facing point clouds of multiple targets with varying sparsity, thus improving the accuracy and speed of target recognition.

[0007] In a first aspect, embodiments of this application provide a target recognition method, the method comprising: The target segmentation image and original point cloud of the environment in which the target robot is located are obtained, and the first pixel points obtained by projecting each original point in the original point cloud onto the pixel coordinate system; the target segmentation image contains the mask region of each target; For any target, a second pixel is set as the first pixel within the mask area of ​​the target; seed points are selected from the original points corresponding to all second pixels based on the first distance between each second pixel and the center point of the mask area of ​​the target and the depth value of the original point corresponding to each second pixel. The adaptive search radius corresponding to the seed point is determined based on the second distance between the seed point and the original point corresponding to each second pixel point; Within the masked area of ​​the target, starting from the seed point, search for the original point corresponding to the second pixel within the adaptive search radius; take the new original point found as the new seed point, and continue searching within the adaptive search radius until there are no new seed points. Based on all the original points found, determine the three-dimensional perception result of the target.

[0008] In one possible implementation, the step of selecting seed points from all the original points corresponding to the second pixels based on the first distance between each second pixel point and the center point of the mask region of the target and the depth value of the original point corresponding to each second pixel point includes: The first preset number of second pixels with the smallest first distance are determined as the fourth pixel; Sort the original points according to the depth values ​​of each fourth pixel, and determine the original point corresponding to the median depth value as the seed point.

[0009] In one possible implementation, determining the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the corresponding original point of each second pixel point includes: The second preset number of second pixels with the smallest second distance are determined as the fifth pixel; The adaptive search radius corresponding to the seed point is determined based on the median of the second distance between the seed point and the original point corresponding to each fifth pixel point.

[0010] In one possible implementation, the adaptive search radius corresponding to the seed point is determined based on the median of the second distances between the seed point and the corresponding original points of each fifth pixel, including: The median of the second distance between the seed point and the original point corresponding to each fifth pixel point is determined as the initial adaptive search radius; Determine whether the value corresponding to the initial adaptive search radius meets the preset radius upper and lower limit constraints; the preset radius upper and lower limit constraints include a preset radius upper limit and a preset radius lower limit; If the preset radius upper and lower limits are met, the initial adaptive search radius is determined as the final adaptive search radius. Otherwise, the initial adaptive search radius, the upper limit of the preset radius, and the lower limit of the preset radius are compared to obtain the final adaptive search radius.

[0011] In one possible implementation, the final adaptive search radius is obtained by comparing the initial adaptive search radius value, the preset upper radius value, and the preset lower radius value using the following formula: ; in, For the final adaptive search radius, This corresponds to the initial adaptive search radius value. This is the value corresponding to the preset lower limit of the radius. This is the value corresponding to the preset upper limit of the radius.

[0012] In one possible implementation, determining the 3D perception result of the target based on all the searched original points includes: Fit the original points found to obtain a 3D bounding box; The three-dimensional bounding box and the category of the target in the target segmentation image are combined to obtain the three-dimensional perception result of the target.

[0013] In one possible implementation, acquiring the target segmentation image of the environment in which the target robot is located includes: The two-dimensional image perceived by the target robot in its environment is input into the target segmentation model to obtain an initial segmentation image; in the initial segmentation image, the mask pixels of the target points are non-zero values, and the mask pixels of points other than the target points are zero values; Convert the initial segmented image into a binary image; Perform image erosion on the binary image; Using the non-zero pixels in the eroded binary image as a reference, the mask pixel values ​​at the corresponding positions in the initial segmentation image are retained, and the mask pixels at other positions are set to 0 to obtain the target segmentation image.

[0014] Secondly, embodiments of this application also provide a target recognition device, the device comprising: The acquisition module is used to acquire the target segmentation image and original point cloud of the environment where the target robot is located, and the first pixel points obtained by projecting each original point in the original point cloud onto the pixel coordinate system; the target segmentation image contains the mask region of each target; The filtering module is used to filter seed points from all the original points corresponding to the second pixels for any target based on the first distance between each second pixel point and the center point of the mask region of the target and the depth value of the original point corresponding to each second pixel point; the second pixel point is the first pixel point within the mask region of the target. The determination module is used to determine the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the original point corresponding to each second pixel point; The region growing and 3D perception result generation module is used to search for original points within an adaptive search radius starting from a seed point within the masked area of ​​the target; the newly found original points are used as new seed points, and the search continues within the adaptive search radius until no new seed points are found; based on all the found original points, the 3D perception result of the target is determined.

[0015] In one possible implementation, the filtering module is specifically used to determine the first preset number of second pixels with the smallest first distance as fourth pixels; sort according to the depth values ​​of the original points corresponding to each fourth pixel, and determine the original point corresponding to the median depth value as a seed point.

[0016] In one possible implementation, the determining module is specifically used to determine the second preset number of second pixels with the smallest second distance as the fifth pixel; and to determine the adaptive search radius corresponding to the seed point based on the median of the second distance between the seed point and the original point corresponding to each fifth pixel.

[0017] In one possible implementation, the determining module is specifically configured to determine the median of the second distance between the seed point and the original point corresponding to each fifth pixel point as the initial adaptive search radius; determine whether the value corresponding to the initial adaptive search radius meets the preset radius upper and lower limit constraints; the preset radius upper and lower limit constraints include a preset radius upper limit and a preset radius lower limit; if it meets the preset radius upper and lower limit constraints, then the initial adaptive search radius is determined as the final adaptive search radius; otherwise, the value corresponding to the initial adaptive search radius, the value corresponding to the preset radius upper limit, and the value corresponding to the preset radius lower limit are compared to obtain the final adaptive search radius.

[0018] In one possible implementation, the determining module is specifically used to compare the initial adaptive search radius corresponding value, the preset radius upper limit corresponding value, and the preset radius lower limit corresponding value using the following formula to obtain the final adaptive search radius: ; in, For the final adaptive search radius, This corresponds to the initial adaptive search radius value. This is the value corresponding to the preset lower limit of the radius. This is the value corresponding to the preset upper limit of the radius.

[0019] In one possible implementation, the region growing and 3D perception result generation module is specifically used to fit all the searched original points to obtain a 3D bounding box; and to combine the 3D bounding box with the category of the target in the target segmentation image to obtain the 3D perception result of the target.

[0020] In one possible implementation, the acquisition module is specifically used to input the two-dimensional image perceived by the target robot in its environment into the target segmentation model to obtain an initial segmentation image; the mask pixels of the target points in the initial segmentation image are non-zero values, and the mask pixels of points other than the target points are zero values; the initial segmentation image is converted into a binary image; image erosion is performed on the binary image; based on the non-zero pixels in the eroded binary image, the mask pixel values ​​of the corresponding positions in the initial segmentation image are retained, and the mask pixels of the remaining positions are set to 0 to obtain the target segmentation image.

[0021] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the target recognition method as described in any of the first aspects.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the target recognition method as described in any of the first aspects.

[0023] This application provides a target recognition method, apparatus, electronic device, and storage medium. The method includes: selecting seed points based on a first distance between a second pixel point obtained from the projection of each original point cloud and the center point of the target's mask region, and the depth value of the original point corresponding to each second pixel point; the second pixel point is located within the mask region of the target in the target segmentation image; adaptive region growing is performed based on adaptive seed points and a search radius to search for the original points corresponding to the second pixels points on the target; and the three-dimensional perception result of the target is determined based on all the searched original points. Compared with the prior art using fixed seed points and fixed radii, the beneficial effects of this application include: 1. The method searches the point clouds of each target and some noise separately, and has the following advantages: (1) The method can adaptively calculate the search radius and find seed points without setting fixed values. (2) The method can simultaneously search and identify dense target point clouds in the vicinity and sparse target point clouds in the distance without adjusting parameters. (3) The method can identify and eliminate noise points without additional clustering of noise points. (4) Only the point cloud of the target that falls within the mask area of ​​a target needs to be clustered, avoiding redundancy in clustering information. (5) The algorithm has the advantages of high accuracy and fast search speed.

[0024] 2. The target recognition method provided by the embodiments of this application is a target adaptive region growth recognition method based on two-dimensional segmentation and point cloud, which realizes three-dimensional perception of targets based on the fusion of two-dimensional segmentation results and point cloud, and can quickly and accurately detect targets in the environment. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a target recognition method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating the acquisition of a target segmentation image provided in an embodiment of this application is shown; Figure 3A flowchart illustrating the projection process of the original point cloud provided in an embodiment of this application is shown; Figure 4 The flowchart of the adaptive search process for seed points provided in the embodiments of this application is shown; Figure 5 The flowchart illustrating the adaptive determination of the search radius provided in an embodiment of this application is shown. Figure 6 The flowchart illustrating the adaptive search radius determination process based on upper and lower limit constraints provided in this application embodiment is shown. Figure 7 A flowchart illustrating the determination of three-dimensional perception results provided in an embodiment of this application is shown; Figure 8 This illustration shows a schematic diagram of the structure of a target recognition device provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0028] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario of "quadruped robot environmental perception," the following implementation methods are provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application primarily describes the "quadruped robot environmental perception field," it should be understood that this is merely an exemplary embodiment.

[0030] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0031] The target recognition method provided in the embodiments of this application will be described in detail below.

[0032] Reference Figure 1 The diagram shown is a flowchart illustrating a target recognition method provided in an embodiment of this application. The exemplary steps of this embodiment are described below: S101. Obtain the target segmentation image and original point cloud of the environment where the target robot is located, and the first pixel point obtained by projecting each original point in the original point cloud onto the pixel coordinate system.

[0033] In this embodiment, when the target robot performs a task (such as object handling, environmental inspection, or path planning), a target segmentation image and an original point cloud of the environment in which the target robot is located are acquired, along with the first pixel points obtained by projecting each original point in the original point cloud onto a pixel coordinate system. The target segmentation image is a grayscale image. In the target segmentation image, different pixel values ​​are assigned to regions of different targets (objects to be identified) in the environment in which the target robot is located. These pixel values ​​constitute the target mask. Each mask region represents a specific target, and the target category information is usually identified through the pixel values ​​of the mask or metadata associated with the mask. Therefore, the target segmentation image contains mask regions for each target, and the corresponding target category is marked in each target mask region.

[0034] For example, assuming the target segmentation image is a grayscale image, with the target region having a pixel value of 235 and the background having a pixel value of 0, then the target segmentation image can be represented as: .

[0035] Specifically, refer to Figure 2 The diagram shown is a flowchart of the process for obtaining a target segmentation image according to an embodiment of this application. Specifically, the target segmentation image of the environment in which the target robot is located is obtained through the following steps: S201. Input the two-dimensional image perceived by the target robot in its environment into the target segmentation model to obtain the initial segmentation image.

[0036] In this embodiment, the two-dimensional image is an image of the surrounding environment captured by a camera mounted on the target robot, used to capture visual information about the environment. These images are typically in color and provide rich visual details, including color, texture, and the relative positions of objects. The target segmentation model is pre-trained based on a large number of two-dimensional sample images and the corresponding segmentation images for each two-dimensional sample image, and is capable of segmenting targets in the image. The initial segmentation image contains mask regions for each target, and the corresponding target category is labeled in each target mask region.

[0037] The target segmentation model can be a deep learning model such as a fully convolutional network (FCN) or U-Net. In the initial segmentation image, the mask pixels for target points are non-zero values, while the mask pixels for points other than target points are zero values. A target point refers to a pixel on the target in the image. S202. Convert the initial segmented image into a binary image.

[0038] In this embodiment of the application, the pixel value of each point within the masked region of each target in the initial segmentation image is set to 255, and the pixel value of points outside the masked region is set to 0, resulting in a binary image, such as... .

[0039] S203. Perform image erosion on the binary image.

[0040] In this embodiment, the erosion operation for grayscale and binary images differs in OpenCV. Specifically, when performing erosion on a binary image in OpenCV, the zero value in the sliding window is used as the center point value. This means that the erosion operation sets the edge pixel values ​​of the target region to zero, thereby reducing the size of the target region. However, when performing erosion on a grayscale image, the minimum value in the sliding window is selected as the center point value. This method reduces the variation in pixel values ​​in the grayscale image.

[0041] The erosion operation on the initial segmentation image follows the same method as the erosion operation on the grayscale image. Directly eroding the initial segmentation image might cause changes in the positions of the mask pixels, thus affecting the accuracy of target recognition. Therefore, this application chooses to perform the erosion operation on the binary image of the initial segmentation image to avoid such positional changes. The detailed erosion process is as follows: For each point in the binary image, if there is a non-target point with a pixel value of 0 (i.e., a point not within the mask area, i.e., not on the target) within a window of a preset size centered on that point, then the pixel value of that point is set to 0; otherwise, the pixel value of that point remains unchanged.

[0042] Example, a binary image is Erosion is performed on a window with a preset size of 3*3. Then, for the point at coordinates (1, 1), the pixel value within the window centered on that point is: Since there are non-target points with a pixel value of 0 within the window, the pixel value of the point at coordinates (1, 1) is set to 0; for the point at coordinates (2, 2), the pixel values ​​within the window centered on that point are: Since there are no non-target points with a pixel value of 0 within the window, the pixel value of the point with coordinates (2, 2) is set to 255.

[0043] S204. Using the non-zero pixels in the eroded binary image as a reference, retain the mask pixel values ​​at the corresponding positions in the initial segmentation image, and set the mask pixels at other positions to 0 to obtain the target segmentation image.

[0044] In this embodiment, a non-zero pixel refers to a pixel in the eroded binary image that is not equal to 0. The mask pixel value of the pixel at the position corresponding to the non-zero pixel in the initial segmentation image remains unchanged; the mask pixel value of the pixel at all other positions in the initial segmentation image except for the positions corresponding to the non-zero pixel is set to 0.

[0045] For example, in the eroded binary image, the only non-zero pixels are (3,3) and (4,5). In the initial segmented image, the mask pixels of the pixels at (3,3) and (4,5) remain unchanged, while the mask pixels of the pixels other than (3,3) and (4,5) are set to 0.

[0046] Here, image erosion is performed on the binary image corresponding to the initial segmentation image, which can effectively reduce the erroneously segmented target mask edge region in the initial segmentation image. This ensures that when the original point cloud is subsequently projected onto the target mask region, the original points corresponding to the second pixel point will contain a majority of original target points and a minority of noisy original points, improving the accuracy of subsequent adaptive radius and adaptive seed point determination, thereby improving the accuracy and robustness of target recognition results.

[0047] Furthermore, the original point cloud is a collection of numerous points in the environment collected by the LiDAR (Light Detection and Ranging) radar mounted on the target robot. Each point contains its three-dimensional coordinates (x, y, z) in the LiDAR coordinate system. The x-coordinate represents the point's position (depth value) on the X-axis, the y-coordinate represents the point's position on the Y-axis, and the z-coordinate represents the point's position on the Z-axis. The pixel coordinate system is a two-dimensional coordinate system used to describe the position of pixels in an image. In the LiDAR coordinate system, with the vehicle's forward movement as the viewpoint, the X-axis is positively oriented horizontally forward, the Y-axis is positively oriented horizontally to the left, and the Z-axis is positively oriented perpendicular to both the X and Y axes.

[0048] Specifically, refer to Figure 3 The diagram shown is a flowchart of the projection process of the original point cloud provided in this embodiment of the application. Specifically, the original points in the original point cloud are projected onto the pixel coordinate system through the following steps to obtain the first pixel: S301. Based on the pose matrix of the LiDAR and camera mounted on the target robot and the intrinsic parameters of the camera mounted on the target robot, project each original point in the original point cloud into the pixel coordinate system to obtain the first pixel point corresponding to each original point.

[0049] In this embodiment of the application, for each original point, the pose matrix of the LiDAR and camera mounted on the target robot, the intrinsic parameters of the camera mounted on the target robot, and the three-dimensional coordinates corresponding to the original point are substituted into the following formula to obtain the first pixel point corresponding to the original point.

[0050] in, These are the three-dimensional coordinates corresponding to the original point. This represents the pose matrix of the LiDAR and camera. Let be a rotation matrix. It is a translation matrix. This is the intrinsic parameter matrix of the camera. This is the camera's focal length in the horizontal direction (in pixels). These are the camera's focal lengths in the vertical direction (in pixels), () represents the coordinates of the camera principal point in the pixel coordinate system. This represents the depth value of the original point in the camera coordinate system. These are the coordinates of the first pixel corresponding to the original point.

[0051] Here, in the camera coordinate system, from the perspective of the vehicle's forward movement, the Z-axis is positive in the horizontal forward direction, the X-axis is positive in the horizontal right direction, and the Y-axis is positive in the downward direction perpendicular to the X-axis and Z-axis.

[0052] S302. The original points within the target segmented image with depth values ​​greater than 0 are determined as the final original points.

[0053] In this embodiment, the original points within the target segmentation image with a depth value greater than 0 in the camera coordinate system are determined as the final original points. By filtering the original points in the original point cloud, only the portion of the point cloud containing the target point cloud within the target segmentation image can be retained, reducing subsequent workload and improving target recognition speed.

[0054] S102. For any target, set the second pixel as the first pixel within the target's mask area; based on the first distance between each second pixel and the center point of the target's mask area and the depth value of the original point corresponding to each second pixel, select seed points from all the original points corresponding to the second pixels.

[0055] In the embodiments of this application, seed points can be adaptively selected for different mask regions based on different regional features such as the first distance of each second pixel point from the center point of the mask region and the depth value of the original point corresponding to each second pixel point, so as to improve the target recognition accuracy.

[0056] The first distance between the second pixel and the center point of the mask area refers to the two-dimensional pixel Euclidean distance between the second pixel and the center point of the mask area.

[0057] Reference Figure 4 The diagram shown is an adaptive search flowchart for seed points provided in an embodiment of this application. Specifically, seed points are selected from the original points corresponding to all second pixels based on the first distance between each second pixel and the center point of the target's mask region and the depth value of the original point corresponding to each second pixel. This includes: S401. The first preset number of second pixels with the smallest first distance are determined as the fourth pixel.

[0058] In this embodiment, the smaller the first distance between the second pixel and the center point of the mask area of ​​the target, the closer the second pixel is to the center of the target. If the number of second pixels is less than or equal to a first preset number, all second pixels are determined as fourth pixels; if the number of second pixels is greater than the first preset number, the first preset number (e.g., 15) second pixels with the smallest first distance are determined as fourth pixels.

[0059] Furthermore, in this embodiment, the center point of the target's mask region is calculated based on the coordinate values ​​of all second pixels. Specifically, the center point of the target's mask region is obtained by substituting the coordinates of all second pixels into the following formula.

[0060] ; in, Let be the coordinates of the i-th second pixel. Let be the horizontal coordinate value of the i-th second pixel. Let be the vertical coordinate value of the i-th second pixel. The number of the second pixel. The coordinates of the center point of the masked area of ​​the target.

[0061] S402. Sort the original points according to the depth values ​​of each fourth pixel, and determine the original point corresponding to the median depth value as the seed point.

[0062] In this embodiment, the original points are sorted according to the depth value of the original point corresponding to each fourth pixel point in ascending order to obtain a first original point sequence; the original points are arranged in order in the first original point sequence; in the first original point sequence, the depth value corresponding to the later original point is greater than or equal to the depth value of the earlier original point.

[0063] For example, the first original point sequence is: Then the first origin point The corresponding depth value is less than or equal to the second origin point. The corresponding depth value; the second origin point The corresponding depth value is less than or equal to the third origin point. The corresponding depth value.

[0064] Specifically, by substituting the depth values ​​corresponding to the original points in the first original point sequence into the following formula, the median depth value is obtained: ; in, The median of the depth values. This represents the depth value of the first original point in the first original point sequence. This represents the depth value of the second original point in the first original point sequence. Let W be the depth value of the Wth original point in the first original point sequence, where W is the total number of original points.

[0065] Here, according to the formula above, when the total number of original points is even (e.g., 2, 4, 6), the number of median depth values ​​is two. When the total number of original points is odd (e.g., 1, 3, 5), the number of median depth values ​​is one.

[0066] Therefore, in one alternative implementation, if there are two target depth values, the original point with the smaller target depth value is taken as the seed point, and there is only one seed point for a perceived target.

[0067] Here, this application embodiment provides an adaptive seed point search method. By using a first distance to determine the seed point, the seed point can be located near the center point of the target's mask region. Furthermore, since the second pixel is the first pixel in the target's mask region, the seed point falls within the target's mask region, allowing the seed point to be as close as possible to the center of the target's two-dimensional plane. Using the point with the median depth value among a first preset number of fourth pixels as the seed point allows the seed point to be as close as possible to the target's three-dimensional center, thus resulting in more points on the target and fewer noise points in the subsequently searched original points. Further, the image erosion process and original point cloud projection process provided in this application embodiment can provide high-quality target segmentation images and original point clouds, bringing advantages such as a large number of target points and a small number of noise points. Taking points corresponding to features such as the median depth value as seed points ensures that the seed points are as close as possible to the target, rather than noise points.

[0068] S103. Determine the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the original point corresponding to each second pixel point.

[0069] In the embodiments of this application, the search radius can be adaptively determined for the point cloud of different targets based on the mask region features, such as the second distance between the original points corresponding to each second pixel point in the mask region of the seed point and the target, thereby improving the target recognition accuracy.

[0070] The second distance between the seed point and the original point corresponding to the second pixel refers to the three-dimensional Euclidean distance between the seed point and the original point corresponding to the second pixel.

[0071] Reference Figure 5 The diagram shown is an adaptive search radius determination flowchart provided in an embodiment of this application. Specifically, the adaptive search radius corresponding to the seed point is determined based on the second distance between the seed point and the original point corresponding to each second pixel point, including: S501. The second preset number of second pixels with the smallest second distance are determined as the fifth pixel.

[0072] In this embodiment, the closer the second distance between the seed point and the corresponding original point of each second pixel point, the closer the original point corresponding to the second pixel point is to the seed point. Therefore, the original point corresponding to the second pixel point is more representative of the distance between the seed point and points on the nearby target. The search radius determined based on the fifth pixel point based on the second distance is more reasonable, thereby improving the quality of subsequently searched original points, minimizing noise points, and improving target recognition accuracy. Specifically, if the number of second pixels is less than or equal to a second preset number, all second pixels are determined as fifth pixels; if the number of second pixels is greater than the second preset number, the second pixels with the smallest second distance (e.g., 100) are determined as fifth pixels.

[0073] S502. Determine the adaptive search radius corresponding to the seed point based on the median of the second distance between the seed point and the original point corresponding to each fifth pixel point.

[0074] In this embodiment, the median of the second distance between the seed point and the corresponding original point of each fifth pixel can be determined as the adaptive search radius corresponding to the seed point. Upper and lower limits can also be imposed on the search radius to constrain the value of the adaptive radius and prevent it from being too large or too small. Furthermore, the second distance in this application can be a three-dimensional Euclidean distance.

[0075] Specifically, refer to Figure 6 The diagram shown is a flowchart of the adaptive search radius determination process based on upper and lower limit constraints provided in an embodiment of this application.

[0076] S601. Determine the initial adaptive search radius by taking the median of the second distance between the seed point and the original point corresponding to each fifth pixel point.

[0077] In this embodiment, the adaptive radius is calculated by sampling the features (second distance) of the fifth pixel in the core region near the seed point, using the previously determined adaptive seed point as the center, which is more representative. The specific process for determining the initial search radius is as follows: Step 1: Sort all fifth pixels in ascending order based on the second distance to obtain the second pixel sequence.

[0078] The second pixel sequence contains sequentially arranged fifth pixels. In this second pixel sequence, the second distance between the seed point and the original point corresponding to the preceding fifth pixel is less than or equal to the second distance between the seed point and the original point corresponding to the following fifth pixel.

[0079] For example, the sequence of the second pixel is Then the seed point and the first fifth pixel point The second distance between the corresponding original points is less than or equal to the distance between the seed point and the second fifth pixel. The second distance between the corresponding original points; the seed point and the second fifth pixel point The second distance between the corresponding original points is less than or equal to the distance between the seed point and the third fifth pixel. The second distance between the original points.

[0080] Step 2: Substitute the second distance between the seed point and the original point corresponding to each fifth pixel in the second pixel sequence into the following formula to obtain the initial search radius corresponding to the seed point.

[0081] ; Where r is the initial search radius corresponding to the seed point. For the operation of taking the median, ( () represents the coordinates of the seed point. Let be the coordinates of the seed point in the horizontal forward direction within the coordinate system. The seed point's horizontal coordinates are located at the top left of the coordinate system. Let the seed point be the coordinates of the point in the vertical upward direction in the coordinate system. () represents the coordinates of the original point corresponding to the i-th fifth pixel in the second pixel sequence. Let be the coordinates of the original point corresponding to the i-th fifth pixel in the second pixel sequence, in the horizontal forward direction of the coordinate system. Let be the coordinates of the original point corresponding to the i-th fifth pixel in the second pixel sequence, in the coordinate system, in the horizontal leftward direction. Let be the coordinates of the original point corresponding to the i-th fifth pixel in the second pixel sequence, in the vertical upward direction of the coordinate system, and D be the number of fifth pixels.

[0082] Here, the adaptive search radius determination method provided in this application uses the median of the second distance as the adaptive search radius. With the help of the image erosion process and the original point cloud projection process provided in this application, high-quality target segmentation images and original point clouds can be provided, which brings advantages such as a large number of target points and a small number of noise points. It can be inferred that the median of the second distance is the result of a certain point on the target and a seed point, so that the adaptive search radius is the distance between a point on the target and a target seed point.

[0083] S602. Determine whether the value corresponding to the initial adaptive search radius meets the preset radius upper and lower limit constraints; the preset radius upper and lower limit constraints include the preset radius upper limit and the preset radius lower limit.

[0084] In this application embodiment, to further improve the robustness of target point cloud clustering, the embodiments of this application design a preset upper radius limit and a preset lower radius limit to constrain the numerical range of the adaptive search radius. If the identified targets are mainly people, vehicles, or two-wheeled vehicles, then the preset upper radius limit and preset lower radius limit are designed in combination with the actual physical size of the targets to constrain the numerical value of the search radius and prevent it from being too large or too small.

[0085] Specifically, if the initial search radius is greater than or equal to the preset lower radius limit and less than the preset upper radius limit, or greater than the preset lower radius limit and less than or equal to the preset upper radius limit, then the value corresponding to the initial search radius meets the preset upper and lower radius limit constraints; if the initial search radius is less than the preset lower radius limit or greater than the preset upper radius limit, then the value corresponding to the initial search radius does not meet the preset upper and lower radius limit constraints.

[0086] S603. If the preset upper and lower radius constraints are met, the initial adaptive search radius will be determined as the final adaptive search radius.

[0087] S604. Otherwise, compare the initial adaptive search radius corresponding value, the preset upper radius corresponding value, and the preset lower radius corresponding value to obtain the final adaptive search radius.

[0088] In this embodiment, the final search radius is obtained by comparing the initial search radius value, the preset upper radius value, and the preset lower radius value using the following formula: ; in, For the final adaptive search radius, This corresponds to the initial adaptive search radius value. This is the value corresponding to the preset lower limit of the radius. This is the value corresponding to the preset upper limit of the radius.

[0089] S104. Within the masked area of ​​the target, starting from the seed point, search for the original point corresponding to the second pixel within the adaptive search radius; take the new original point found as the new seed point, and continue searching according to the adaptive search radius until there are no new seed points. Based on all the original points found, determine the three-dimensional perception result of the target.

[0090] In this embodiment, a new original point refers to an original point that has not been searched as a seed point. In the point cloud formed by the original points corresponding to the second pixel of the target's mask region, starting from the adaptive seed point obtained in S102, the search is performed on the original points corresponding to the second pixel within the adaptive search radius obtained in step S103; that is, the search is performed on the original points whose second distance from the seed point is less than the value corresponding to the search radius. The searched original points are recorded as points on the target. The newly searched original points are then used as new seed points, and an adaptive search radius is determined for them according to the method in step S103. The search continues until no new original points are found, thus achieving region growth recognition of the target's point cloud and the removal of noise points. Then, based on all the searched original points, the 3D perception result of the target is determined.

[0091] For example, the point cloud formed by the original points corresponding to the second pixel in the masked area of ​​the target includes the original points corresponding to the second pixel A, B, C, D, E, F, etc. S102 filters out original point A as a seed point, and S103 determines the adaptive search radius R. Then, starting from seed point A, the search continues for the original points corresponding to the second pixel within the adaptive search radius R, obtaining original points A, D, and F corresponding to the second pixel whose distance from seed point A is less than the adaptive search radius R. D and F are new original points, so they are used as seed points to continue the search. Starting from seed point D, the search continues for the original points corresponding to the second pixel within the adaptive search radius R, obtaining original points A and D corresponding to the second pixel whose distance from seed point D is less than the search radius R. A and D are not new original points, so no new seed points are found. Starting from seed point F, the search continues for the original point corresponding to the second pixel within the adaptive search radius R. This yields original points B and F corresponding to the second pixel whose distance from seed point F is less than the adaptive search radius R. Since B is a new original point, it is used as the seed point to continue the search. Starting from seed point B, the search continues for the original point corresponding to the second pixel within the adaptive search radius R. This yields original points B and F corresponding to the second pixel whose distance from seed point B is less than the adaptive search radius R. Since B and F are not new original points, no new seed points are found, and the search is stopped. Finally, based on all the original points A, B, D, and F corresponding to the found second pixel, the 3D perception result of the target is determined.

[0092] Furthermore, referring to Figure 7 The diagram shown is a flowchart illustrating the determination of 3D perception results provided in this embodiment of the application. Based on all the searched original points, the 3D perception results of the target are determined, including: S701. Fit all the original points found to obtain a 3D bounding box.

[0093] In this embodiment, all the searched original points are obtained by performing region growing on the original points corresponding to all second pixels based on the previously determined adaptive seed points and adaptive search radius. The point cloud formed by all the searched original points is the target point cloud, which contains the target's position information in three-dimensional space. These original points are processed using Principal Component Analysis (PCA) to obtain a three-dimensional bounding box (i.e., a 3D bounding box), which accurately encloses the target and provides information on the target's position, size, and orientation in three-dimensional space.

[0094] S702. Combine the 3D bounding box and the target category in the target segmentation image to obtain the 3D perception result of the target.

[0095] In this embodiment, each target mask region in the target segmentation image is labeled with its category (e.g., "vehicle", "pedestrian", etc.). The fitted 3D bounding box is combined with the target's category to generate a complete 3D perception result. This result includes not only the target's 3D geometric information (position, size, orientation) but also its category information, enabling the robot to more comprehensively understand target objects in the environment.

[0096] Furthermore, regarding the target recognition method provided in this application, the point cloud of the target (i.e., the original points corresponding to all the searched second pixel points) is obtained by performing a region growing search on the point cloud formed by points falling within the mask region of the target, based on an adaptive search radius (refer to S103) and an adaptive seed point (refer to S102), in order to identify the target's point cloud and filter out noise point clouds. Compared with the prior art, the beneficial effects of the embodiments of this application include: 1. When the point cloud contains only one target and some noise, the method has the following advantages: (1) The method can adaptively calculate the search radius and find seed points without setting fixed values. (2) The method can simultaneously search and identify dense target point clouds in the vicinity and sparse target point clouds in the distance without adjusting parameters. (3) The method can identify and exclude noise points without additional clustering of noise points. (4) Only the point cloud of the target that falls within the mask area of ​​a target needs to be clustered, avoiding redundancy in clustering information. (5) The algorithm has the advantages of high accuracy and fast search speed.

[0097] 2. The target recognition method provided by the embodiments of this application is a target adaptive region growth recognition method based on two-dimensional segmentation and point cloud, which realizes three-dimensional perception of targets based on the fusion of two-dimensional segmentation results and point cloud, and can quickly and accurately detect targets in the environment.

[0098] Based on the same inventive concept, this application also provides a target recognition device corresponding to the target recognition method. Since the principle of the device in this application is similar to that of the target recognition method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0099] Reference Figure 8 The diagram shown is a schematic representation of a target recognition device provided in an embodiment of this application. The target recognition device includes: The acquisition module 801 is used to acquire the target segmentation image and the original point cloud of the environment where the target robot is located, and the first pixel points obtained by projecting each original point in the original point cloud onto the pixel coordinate system; the target segmentation image contains the mask region of each target; The filtering module 802 is used to set a second pixel as a first pixel within the mask area of ​​the target for any target; and to filter seed points from the original points corresponding to all second pixels based on the first distance between each second pixel and the center point of the mask area of ​​the target and the depth value of the original point corresponding to each second pixel. The determining module 803 is used to determine the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the original point corresponding to each second pixel point; The region growing and 3D perception result generation module 804 is used to search for the original point corresponding to the second pixel point within the mask area of ​​the target, starting from the seed point and within an adaptive search radius; take the newly found original point corresponding to the second pixel point as the new seed point, and continue searching within the adaptive search radius until there are no new seed points; and determine the 3D perception result of the target based on all the original points corresponding to the second pixel points found.

[0100] This application provides a target recognition device that can obtain adaptive seed points and search radii based on features in the mask regions of different targets, thereby recognizing the targets, meeting the search requirements of different targets, and improving the accuracy and speed of target recognition.

[0101] like Figure 9 As shown in the embodiment of this application, an electronic device 900 includes a processor 901, a memory 902, and a bus. The memory 902 stores machine-readable instructions executable by the processor 901. When the electronic device is running, the processor 901 communicates with the memory 902 via the bus, and the processor 901 executes the machine-readable instructions to perform the steps of the target recognition method described above.

[0102] Specifically, the memory 902 and processor 901 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 901 runs the computer program stored in the memory 902, it can execute the target recognition method mentioned above.

[0103] Corresponding to the above target recognition method, this application embodiment also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform the steps of the above target recognition method.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0105] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0107] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the target recognition method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0108] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A target recognition method, characterized in that, The method includes: The target segmentation image and original point cloud of the environment in which the target robot is located are obtained, and the first pixel points obtained by projecting each original point in the original point cloud onto the pixel coordinate system; the target segmentation image contains the mask region of each target; For any target, a second pixel is set as the first pixel within the mask area of ​​the target; seed points are selected from the original points corresponding to all second pixels based on the first distance between each second pixel and the center point of the mask area of ​​the target and the depth value of the original point corresponding to each second pixel. The adaptive search radius corresponding to the seed point is determined based on the second distance between the seed point and the original point corresponding to each second pixel point; Within the masked area of ​​the target, starting from the seed point, search for the original point corresponding to the second pixel within the adaptive search radius; take the new original point found as the new seed point, and continue searching according to the adaptive search radius until there are no new seed points. Based on all the original points found, determine the three-dimensional perception result of the target. The step of selecting seed points from all the original points corresponding to the second pixels based on the first distance between each second pixel point and the center point of the mask region of the target and the depth value of the original point corresponding to each second pixel point includes: determining the first preset number of second pixels points with the smallest first distance as fourth pixels; sorting according to the depth value of the original points corresponding to each fourth pixel point, and determining the original point corresponding to the median depth value as a seed point. The step of determining the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the original point corresponding to each second pixel point includes: determining the second preset number of second pixels with the smallest second distance as the fifth pixel point; and determining the adaptive search radius corresponding to the seed point based on the median of the second distances between the seed point and the original point corresponding to each fifth pixel point.

2. The target recognition method according to claim 1, characterized in that, The adaptive search radius corresponding to the seed point is determined based on the median of the second distance between the seed point and the original point corresponding to each fifth pixel point, including: The initial adaptive search radius is determined by the median of the second distance between the seed point and the original point corresponding to each fifth pixel point. Determine whether the value corresponding to the initial adaptive search radius meets the preset radius upper and lower limit constraints; the preset radius upper and lower limit constraints include a preset radius upper limit and a preset radius lower limit; If the preset radius upper and lower limits are met, the initial adaptive search radius is determined as the final adaptive search radius. Otherwise, the initial adaptive search radius, the upper limit of the preset radius, and the lower limit of the preset radius are compared to obtain the final adaptive search radius.

3. The target recognition method according to claim 2, characterized in that, The final adaptive search radius is obtained by comparing the initial adaptive search radius value, the preset upper radius value, and the preset lower radius value using the following formula: ； in, For the final adaptive search radius, This corresponds to the initial adaptive search radius value. This is the value corresponding to the preset lower limit of the radius. This is the value corresponding to the preset upper limit of the radius.

4. The target recognition method according to claim 1, characterized in that, The determination of the 3D perception result of the target based on all the searched original points includes: Fit the original points found to obtain a 3D bounding box; The three-dimensional bounding box and the category of the target in the target segmentation image are combined to obtain the three-dimensional perception result of the target.

5. The target recognition method according to claim 1, characterized in that, The acquisition of the target segmentation image of the environment in which the target robot is located includes: The two-dimensional image perceived by the target robot in its environment is input into the target segmentation model to obtain an initial segmentation image; in the initial segmentation image, the mask pixels of the target points are non-zero values, and the mask pixels of points other than the target points are zero values; Convert the initial segmented image into a binary image; Perform image erosion on the binary image; Using the non-zero pixels in the eroded binary image as a reference, the mask pixel values ​​at the corresponding positions in the initial segmentation image are retained, and the mask pixels at other positions are set to 0 to obtain the target segmentation image.

6. A target recognition device, characterized in that, The apparatus is used to perform the steps of the target recognition method as described in any one of claims 1 to 5, the apparatus comprising: The acquisition module is used to acquire the target segmentation image and original point cloud of the environment where the target robot is located, and the first pixel points obtained by projecting each original point in the original point cloud onto the pixel coordinate system; the target segmentation image contains the mask region of each target; The filtering module is used to filter seed points from all the original points corresponding to the second pixels for any target based on the first distance between each second pixel point and the center point of the mask region of the target and the depth value of the original point corresponding to each second pixel point; the second pixel point is the first pixel point within the mask region of the target. The determination module is used to determine the adaptive search radius corresponding to the seed point based on the second distance between the seed point and the original point corresponding to each second pixel point; The region growing and 3D perception result generation module is used to search for the original point corresponding to the second pixel point within the mask area of ​​the target, starting from the seed point and within an adaptive search radius; the newly found original point is used as the new seed point, and the search continues according to the adaptive search radius until there are no new seed points; based on all the found original points, the 3D perception result of the target is determined.

7. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the target recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the target recognition method as described in any one of claims 1 to 5.

Images