Point cloud-based target detection method, device, equipment, medium and product

Abstract

The disclosure provides a set of point cloud-based target detection methods, devices, equipment, media and products in the field of artificial intelligence, specifically in the fields of computer vision, virtual reality, deep learning, large models, etc., which can be applied to scenarios such as automatic driving. The specific implementation scheme is: obtaining a detection box obtained from a 3D point cloud, the detection box being a detection box enclosing a target object in the 3D point cloud; based on a plurality of labeled boxes in a training set, at least one target labeled box matched with the detection box is screened; and the detection box is verified according to at least one target labeled box, and a target detection box is obtained.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large model and other technical fields, and can be applied to scenarios such as autonomous driving. In particular, it relates to a point cloud-based target detection method, device, equipment, medium and product. Background Technology

[0002] A 3D point cloud is a collection of point cloud data used to represent spatial distribution under the same spatial reference. The main application of 3D point clouds is target detection, which can be applied to scenarios such as autonomous driving, game localization, robot perception, and 3D reconstruction.

[0003] Currently, it's possible to detect bounding boxes of objects from 3D point clouds, such as detecting bounding boxes containing vehicles within a 3D point cloud; these bounding boxes can be cuboids. However, the bounding boxes obtained from 3D point clouds are not accurate enough and lack precision. Summary of the Invention

[0004] This disclosure provides a point cloud-based target detection method, apparatus, equipment, medium, and product.

[0005] According to a first aspect of this disclosure, a point cloud-based target detection method is provided, comprising:

[0006] Obtain a detection box from a 3D point cloud, wherein the detection box is a detection box that surrounds the target object in the 3D point cloud;

[0007] Based on multiple bounding boxes in the training set, at least one target bounding box that matches the detection box is selected;

[0008] The detection box is validated based on at least one of the target bounding boxes to obtain the target detection box.

[0009] According to a second aspect of this disclosure, a point cloud-based target detection device is provided, comprising:

[0010] An acquisition unit is used to acquire a detection box obtained from a 3D point cloud, wherein the detection box is a detection box that surrounds a target object in the 3D point cloud;

[0011] A filtering unit is used to filter at least one target bounding box that matches the detection box based on multiple bounding boxes in the training set.

[0012] A correction unit is used to perform verification processing on the detection box based on at least one of the target bounding boxes to obtain the target detection box.

[0013] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the first aspect and various possible methods described in the first aspect.

[0017] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the first aspect and various possible methods described in the first aspect.

[0018] According to a fifth aspect of this disclosure, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the method described in the first aspect.

[0019] According to the technical solution of this disclosure, detection boxes obtained from 3D point clouds can be acquired. These detection boxes can be bounding boxes surrounding target objects in the 3D point cloud, achieving preliminary detection of the target objects. Subsequently, based on multiple bounding boxes in the training set, at least one target bounding box matching the detection box can be selected. Then, the detection box is validated based on at least one target bounding box to obtain the target detection box. The at least one target bounding box originates from multiple bounding boxes in the training set, reflecting the true detection boxes in the training set and accurately representing the object's location. Validating the detection box using at least one target bounding box makes the target detection box closer to the true bounding box of the target object, effectively improving the prediction accuracy and precision of the target detection box.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0022] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0023] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0024] Figure 3a A schematic diagram of a detection frame is shown;

[0025] Figure 3b A schematic diagram of a label box is shown;

[0026] Figure 4 This is a schematic diagram according to the third embodiment of the present disclosure;

[0027] Figure 5 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0028] Figure 6 This is an application scenario diagram of a point cloud-based target detection method provided according to an embodiment of this disclosure;

[0029] Figure 7 This is a schematic diagram according to the fifth embodiment of the present disclosure;

[0030] Figure 8 This is a block diagram of an electronic device used to implement the point cloud-based target detection method of the present disclosure embodiments. Detailed Implementation

[0031] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] This disclosure provides a point cloud-based target detection method, apparatus, device, medium, and product that can be applied to the field of artificial intelligence technology, specifically computer vision, virtual reality, deep learning, large model, and other technical fields. It can be applied to scenarios such as autonomous driving to achieve accurate identification of the target object's detection box.

[0033] In related technologies, 3D point cloud is a collection of point cloud data used to represent spatial distribution under the same spatial reference. 3D point cloud data can provide three-dimensional spatial data. Through 3D point clouds, the road environment surrounding a vehicle can be perceived, target detection can be performed, and target tracking can be achieved. Currently, the main application scenario for point cloud data is to identify bounding boxes surrounding target objects from 3D point clouds, and then use these bounding boxes for target object tracking, trajectory recognition and planning, obstacle avoidance, and other processing. Commonly used target object detection algorithms generally use deep learning models to identify bounding boxes surrounding target objects from 3D point clouds. This detection method is relatively coarse, and the obtained bounding boxes deviate from the actual area where the target object is located in terms of size and orientation, resulting in low detection accuracy.

[0034] To address the aforementioned issues, the technical solution of this disclosure optimizes the detection boxes after obtaining them, resulting in more accurate detection boxes. To solve the detection box optimization problem, the technical solution of this disclosure considers that multiple labeled boxes exist in the training set. The detection boxes can be validated using target labeled boxes that match the detection boxes from these multiple labeled boxes. This ensures that the detection box correction process references at least one target labeled box belonging to the ground truth, resulting in target detection boxes that are closer to the ground truth boxes, thus achieving accurate detection box correction.

[0035] The technical solution disclosed herein can obtain detection boxes from 3D point clouds. These detection boxes can be bounding boxes surrounding target objects in the 3D point cloud, achieving preliminary detection of the target objects. Subsequently, based on multiple bounding boxes in the training set, at least one target bounding box matching the detection box can be selected. The detection box is then validated based on this at least one target bounding box to obtain the target detection box. The at least one target bounding box originates from multiple bounding boxes in the training set, reflecting the true detection boxes in the training set and accurately representing the object's location. Validating the detection box using this at least one target bounding box makes the target detection box closer to the true bounding box of the target object, effectively improving the prediction accuracy and precision of the target detection box.

[0036] The technical solution of this application will now be described in detail with reference to the accompanying drawings.

[0037] Figure 1 This is a schematic diagram of the first embodiment of the present disclosure, with reference to... Figure 1The diagram shows a system schematic for implementing a point cloud-based target detection method. The system may include an electronic device 1 and a vehicle 2. A 3D scanning device 3 may be mounted on the roof of the vehicle 2. This 3D scanning device could be, for example, a LiDAR, an RGB (Red, Green, Blue) binocular camera, a 3D structured light camera, or a ToF (Time-of-Flight) camera. While the vehicle 2 is in motion, the 3D scanning device 3 can acquire 3D point clouds in front of the vehicle 2 and transmit these point clouds to the electronic device 1.

[0038] Electronic device 1 can receive 3D point clouds and obtain detection boxes from them. These detection boxes can be bounding boxes that enclose target objects within the 3D point cloud. Then, based on multiple labeled boxes in the training set, at least one target labeled box that matches the detection boxes is selected. This at least one target labeled box can be a ground truth box from the training data. Furthermore, the detection boxes are validated using this at least one target labeled box to obtain target detection boxes. This process makes the detection boxes closer to the ground truth boxes of the target objects, effectively improving the prediction accuracy and precision of the target detection boxes.

[0039] Figure 2 This is a schematic diagram of the second embodiment of the present disclosure, with reference to... Figure 2 The point cloud-based target detection method shown can be configured as a point cloud-based target detection device, which can be located in an electronic device. The point cloud-based target detection method may include the following steps:

[0040] 201. Obtain the detection box from the 3D point cloud. The detection box is the detection box that surrounds the target object in the 3D point cloud.

[0041] In this embodiment, 3D point cloud refers to data represented in three-dimensional space, which can be composed of multiple point cloud data in three-dimensional space. 3D point cloud can be generated by LiDAR scanning or other 3D scanning equipment, and can be applied to technologies such as autonomous driving, augmented reality, and computer vision.

[0042] Prior to step 201, the method further includes receiving a 3D point cloud sent by a 3D scanning device. The 3D scanning device may include, for example, a LiDAR or other 3D scanning equipment.

[0043] Optionally, obtaining the detection boxes from the 3D point cloud includes: detecting the detection boxes in the 3D point cloud using an object detection algorithm to obtain the detection boxes in the 3D point cloud. The object detection algorithm can be any of the neural network algorithms or machine learning algorithms used for 3D point cloud detection, such as the PointPillars algorithm, the Deep3Dbox detection algorithm, etc. In this embodiment, the specific type of object detection algorithm is not limited.

[0044] In this embodiment, the detection box can be a three-dimensional cuboid, and the annotation box can have the same spatial dimension as the detection box. The annotation box can also be a three-dimensional cuboid.

[0045] 202. Based on multiple bounding boxes in the training set, filter at least one target bounding box that matches the detection box.

[0046] Optionally, the detection box can be associated with the category of the target object. Taking the field of autonomous driving as an example, the category of the target object can include any one of vehicles, streetlights, pedestrians, buildings, etc.

[0047] The training set can include multiple training data sets, each of which can include at least one bounding box. These bounding boxes are ground truth boxes that surround objects within the training data and can be obtained through manual or automatic annotation. Each bounding box can be associated with a specific object category.

[0048] To achieve more effective optimization when multiple bounding boxes participate in the prediction of the target object in step 202, multiple bounding boxes with the same object category as the target object can be obtained from the training set before step 202. For example, if the target object is a vehicle, the object category labeled by multiple bounding boxes can all be "vehicle".

[0049] Optionally, step 202 may include: based on multiple bounding boxes in the training set, filtering at least one target bounding box that satisfies the matching conditions corresponding to the detection box.

[0050] 203. Verify the detection boxes based on at least one target bounding box to obtain the target detection boxes.

[0051] Optionally, step 203 may include: using detection boxes to perform correction processing on at least one target bounding box to obtain at least one corrected bounding box. From the at least one corrected bounding box and the detection boxes, determine the target detection boxes that satisfy the verification condition. The verification condition may refer to the maximum number of point clouds contained in the detection box or the corrected bounding box. The number of point clouds may refer to the number of points enclosed by the detection box or the corrected bounding box in the 3D point cloud.

[0052] The target detection box can be a cuboid obtained by verifying the detection box.

[0053] In this embodiment, after step 203, the above-mentioned point cloud-based target detection method may further include: performing position tracking processing on the target object according to the target detection box, obtaining the motion trajectory of the target object, and outputting the motion trajectory of the target object.

[0054] The method may further include: performing navigation processing on the target object based on the target detection box, obtaining the driving path of the target object, generating a navigation electronic map based on the driving path, and outputting the navigation electronic map to facilitate vehicle driving.

[0055] Furthermore, after obtaining the target object's driving path, the target object can be controlled to drive automatically along that path.

[0056] The above application scenarios provide simple examples of the application of object detection boxes in the fields of autonomous driving and transportation, demonstrating how object detection boxes can be widely used.

[0057] The technical solution disclosed herein can obtain detection boxes from 3D point clouds. These detection boxes can be bounding boxes surrounding target objects in the 3D point cloud, achieving preliminary detection of the target objects. Subsequently, based on multiple bounding boxes in the training set, at least one target bounding box matching the detection box can be selected. The detection box is then validated using this at least one target bounding box to obtain the target detection box. The at least one target bounding box originates from multiple bounding boxes in the training set, reflecting the true detection boxes in the training set and accurately representing the object's location. Validating the detection box using this at least one target bounding box makes the target detection box closer to the true bounding box of the target object, effectively improving the prediction accuracy and precision of the target detection box.

[0058] To help readers better understand the implementation principles of this disclosure, the following diagram (Figure 3-) is provided. Figure 6 right Figure 2 The illustrated embodiments are further refined.

[0059] As one embodiment, verifying the detection box based on at least one target bounding box to obtain the target detection box may include:

[0060] At least one target bounding box is transformed into a correction box that matches the geometric information of the detection box, thereby obtaining at least one correction box;

[0061] The target detection box is determined based on at least one calibration box and one detection box.

[0062] In this embodiment, each box has geometric information, which refers to the information constituted by the geometric parameters of that box. Geometric parameters may include, for example, at least one of the following parameters:

[0063] Parameters include dimensions (length, width, height), shape, orientation, and center point coordinates. Orientation can be represented by angles such as heading angle, which refers to the yaw angle of the detection frame on the ground plane.

[0064] At least one bounding box is initially obtained from multiple bounding boxes. The geometric information of this at least one bounding box may differ from that of the detection box. Directly anchoring the target object using the bounding box does not accurately delineate the area where the target object is located. Therefore, the geometric information of the detection box can be used to perform a geometric transformation on the target bounding box to obtain a correction box, ensuring that the geometric information of the correction box is consistent with that of the detection box.

[0065] For example, Figure 3a A detection box 301 is shown. Figure 3b A label box 302 is shown. Figure 3a The detection box 301 shown is Figure 3b The orientation and size of the label box 302 shown are different.

[0066] In this embodiment, the transformation step of the target bounding box may include: calculating the mapping relationship between the geometric information of the target bounding box and the geometric information of the detection box, and transforming the target bounding box into a correction box that matches the geometric information of the detection box according to the mapping relationship.

[0067] Optionally, determining the target detection box based on at least one calibration box and one detection box may include: selecting the target detection box from at least one calibration box and one detection box.

[0068] Furthermore, selecting a target detection box from at least one calibration box and a detection box may include: using at least one calibration box and a detection box as multiple candidate boxes, evaluating the detection quality of each candidate box, obtaining the quality detection results corresponding to each candidate box, and selecting the box with the highest quality detection result as the target detection box.

[0069] The process of evaluating the detection quality of multiple candidate boxes and obtaining the quality detection results corresponding to each candidate box may include: evaluating the quality of the enclosing effect of each candidate box on the target object and obtaining the quality evaluation results corresponding to each candidate box.

[0070] The technical solution disclosed herein uses the geometric information of the detection box when transforming the target bounding box into a correction box, so that the geometric information of the correction box and the detection box are matched. Therefore, when the target detection box is determined based on at least one correction box and a detection box, the selection space of the target detection box is increased. Compared with directly using the detection box as the prediction result, more optional boxes can be provided, and more accurate boxes can be selected as the target detection boxes, effectively improving the accuracy of the target detection boxes.

[0071] Figure 4 This is a schematic diagram of a third embodiment of the present disclosure. The difference from the preceding embodiments is that, based on at least one correction box and a detection box, determining the target detection box includes:

[0072] 401. Based on at least one correction box and one detection box, determine at least one candidate box.

[0073] In one possible design, step 401 may include: determining at least one correction box and a detection box as at least one candidate box.

[0074] In another possible design, step 401 may further include: combining at least one calibration box and a detection box with a 3D point cloud to generate a selection page, the selection page including the 3D point cloud and at least one calibration box and a detection box displayed on the 3D point cloud. In response to a box selection operation performed by the user on at least one calibration box and a detection box, at least one candidate box selected by the user is obtained.

[0075] In another possible design, step 401 may further include: determining at least one candidate bounding box that meets the usage conditions from at least one correction bounding box and at least one detection bounding box. The usage conditions may include the number of points in the bounding area of ​​the 3D point cloud corresponding to the correction bounding box or the detection bounding box being greater than or equal to a number threshold. The number threshold can be set according to the usage conditions; the larger the number threshold, the more points the correction bounding box or the detection bounding box contains within the bounding area of ​​the 3D point cloud, and the more accurate the shape of the target object formed by aggregation.

[0076] 402. Determine the number of point clouds corresponding to at least one candidate box in the 3D point cloud, where the number of point clouds is the number of point cloud data contained in the candidate box in the 3D point cloud.

[0077] In this embodiment, the step of determining the number of point clouds of any candidate box in step 402 may include: acquiring at least one point cloud data in the 3D point cloud, determining the target point cloud data whose coordinates are within the coordinate range of the candidate box based on the coordinates corresponding to the at least one point cloud data, calculating the number of point clouds of the target point cloud data, and obtaining the number of point clouds corresponding to the candidate box in the 3D point cloud.

[0078] Point cloud data refers to the data associated with data points in a 3D point cloud. The coordinates of point cloud data can be, for example, the (x, y, z) coordinates of the data points. The coordinate range of a candidate box means that the (x, y, z) coordinates of the data point belong to the coordinate range of the candidate box. The coordinate range of the candidate box can refer to the area of ​​the cuboid formed by the candidate box.

[0079] 403. Based on the number of point clouds corresponding to at least one candidate box, determine the candidate box corresponding to the largest number of point clouds as the target detection box.

[0080] Optionally, step 403 may include: determining the maximum number of point clouds from the point cloud data corresponding to at least one candidate box, and using the candidate box corresponding to the maximum number of point clouds as the target detection box.

[0081] The maximum number of point clouds can be the maximum value among the point cloud numbers corresponding to at least one candidate box.

[0082] In the technical solution of this disclosure, in order to determine the target detection box from at least one calibration box and one detection box, the at least one calibration box and one detection box can be managed uniformly to obtain at least one candidate box. The point cloud count of each candidate box can be obtained, and the point cloud count represents the detection quality of the candidate box, thus making the quality evaluation data of the candidate box more effective. Therefore, using the candidate box with the largest point cloud count as the target detection box can obtain the target detection box with the highest detection quality, achieving high-precision detection of the target box.

[0083] As an optional implementation, at least one target bounding box is transformed into a correction box that matches the geometric information of the detection box to obtain at least one correction box, including:

[0084] The size of the detection box is adjusted according to the original proportion of the detection box to obtain a standard box, and a set of standard point clouds belonging to the standard box is extracted from the 3D point cloud. The size of the standard box is a preset multiple of the size of the detection box.

[0085] Based on the training data corresponding to at least one target bounding box, extract the set of labeled point clouds corresponding to at least one target bounding box. The training data is the training 3D point cloud that participates in the training, and the training data is associated with the bounding boxes.

[0086] Based on the point cloud set corresponding to at least one target bounding box, and combined with the standard point cloud set, the at least one target bounding box is transformed into a correction box, thereby obtaining at least one correction box.

[0087] Optionally, the original proportions of the detection frame may include its aspect ratio and width-to-height ratio. The aspect ratio can refer to the ratio of the length to the width of the detection frame. The width-to-height ratio can refer to the ratio of the width to the height of the detection frame.

[0088] The process of adjusting the size of the detection frame according to its original proportions to obtain a standard frame can include proportionally adjusting the length, width, and height of the detection frame to obtain the adjusted standard frame. The size of the standard frame is a preset multiple of the size of the detection frame. The preset multiple can be a pre-set multiple or a constant set according to usage requirements, such as 1.2, 1.5, or 2. To avoid large detection errors or distortions, the value of the preset multiple can be within a certain range, for example, greater than a first value and less than a second value. The first and second values ​​can also be set according to usage requirements. For example, the first value can be 1 and the second value can be 2. Of course, this method of setting values ​​is only exemplary and does not constitute a specific limitation on the embodiments of this disclosure.

[0089] According to the original proportions of the detection frame, the length, width, and height of the detection frame are adjusted proportionally. This can include: calculating the product of the length of the detection frame and the preset multiple to obtain the length of the standard frame; calculating the product of the width of the detection frame and the preset multiple to obtain the width of the standard frame; and calculating the product of the height of the detection frame and the preset multiple to obtain the height of the standard frame. The standard frame is then determined by its length, width, and height.

[0090] After adjusting the size of the detection frame according to its original proportions to obtain a standard frame, the process also includes adjusting the center point of the standard frame so that it is the same as the center point of the detection frame, and adjusting the direction of the standard frame so that it is the same as the direction of the detection frame.

[0091] The original scale of the detection bounding box is the same as that of the standard bounding box. Specifically, the original scale of the detection bounding box being the same as that of the standard bounding box can mean that the length-to-width ratio of the standard bounding box is the same as that of the detection bounding box, and the width-to-height ratio of the standard bounding box is the same as that of the detection bounding box. Thus, the standard bounding box inherits the geometric characteristics of the detection bounding box, achieving proportional scaling. The resulting standard bounding box can contain more point cloud data while maintaining the geometric characteristics of the detection bounding box, enabling the standard bounding box to more effectively enclose the target object.

[0092] In this embodiment, during the process of adjusting the detection box to the standard box, the angle and position of the detection box and the standard box are kept consistent, so that the standard box is only enlarged without updating its angle and position, ensuring that the standard box can be accurately positioned to achieve a more accurate enclosure of the target object.

[0093] Optionally, the training set may include multiple training data sets, which can be training 3D point clouds used in the training process. Training 3D point clouds can also refer to point clouds obtained through scanning with a 3D scanning device. Data training can be associated with at least one bounding box. Target bounding boxes can correspond to training data.

[0094] In this embodiment, the step of obtaining the set of labeled point clouds of the target labeled box may include: determining the coordinate range of the target labeled box, determining the coordinates corresponding to multiple training point cloud data in the training data, and determining the training point cloud data whose coordinates are within the coordinate range of the target labeled box as the set of labeled point clouds of the target labeled box.

[0095] Optionally, based on the point cloud set corresponding to at least one target bounding box and the standard point cloud set, the at least one target bounding box is transformed into a correction box to obtain at least one correction box. This may include: extracting the point cloud mapping relationship between the point cloud set and the standard point cloud set to obtain the point cloud mapping relationship corresponding to at least one target bounding box, and converting the target bounding box into a correction box based on the point cloud mapping relationship to obtain the correction box corresponding to at least one target bounding box.

[0096] Furthermore, the point cloud mapping relationship can include the affine transformation matrix between the labeled point cloud set and the standard point cloud set.

[0097] In the technical solution disclosed herein, during the target bounding box correction process, the detection boxes can first be enlarged N times according to their original aspect ratio to obtain standard boxes. These standard boxes contain more point cloud data, effectively enclosing the target object. Then, by defining the point cloud sets of the standard boxes and the point cloud sets of at least one target bounding box, the mapping relationship between points can be obtained. This mapping is then used to map the boxes together. That is, the mapping relationship between points in the point cloud sets is applied to the mapping between boxes, achieving geometric correction from the target bounding box to the correction box, obtaining a correction box that matches the geometric features of the standard box. Since the standard box retains the geometric information of the detection boxes, the correction box possesses the geometric information of the detection boxes enclosing the target object. The correction box can serve as a detection result for the target object, and at least one correction box can participate in the selection of the target detection box, further improving the selection range and obtaining target detection boxes with higher detection accuracy.

[0098] As one embodiment, based on the set of labeled point clouds corresponding to at least one target bounding box, and combined with a set of standard point clouds, the at least one target bounding box is transformed into a correction box to obtain at least one correction box, including:

[0099] Using an affine transformation algorithm, calculate the affine transformation matrix that maps at least one set of labeled point clouds to a set of standard point clouds.

[0100] Affine transformation is performed on the corresponding target bounding boxes based on the affine transformation matrix of the labeled point cloud set to obtain at least one correction box corresponding to each target bounding box.

[0101] Optionally, the affine transformation algorithm can refer to a spatial transformation algorithm determined by linear transformation and translation. Calculating the affine transformation matrix mapping at least one set of labeled point clouds to a standard point cloud set using the affine transformation algorithm can include: constructing an unknown affine transformation matrix; and using the matrix transformation relationship between at least one set of labeled point clouds, the standard point cloud set, and the unknown affine transformation matrix to solve for the unknown affine transformation matrix for each set of labeled point clouds to the standard point cloud set, thereby obtaining the affine transformation matrix corresponding to at least one set of labeled point clouds.

[0102] An affine transformation matrix can include a matrix A and an additional column b, where matrix A can be used for linear transformations and column b can be used for translation transformations. In this embodiment, the affine transformation matrix can be solved using a set of labeled point clouds and a set of standard point clouds.

[0103] Optionally, the set of labeled point clouds can be associated with an affine transformation matrix and a target bounding box. Performing an affine transformation on the corresponding target bounding box based on the affine transformation matrix of the labeled point cloud set can include: calculating the affine transformation of the labeled point cloud set onto the target bounding box corresponding to the labeled point cloud set using the affine transformation matrix, to obtain the corrected bounding box corresponding to the target bounding box. The corrected bounding box can be the corrected bounding box of the target bounding box.

[0104] In the technical solution disclosed herein, an affine transformation algorithm can be used to calculate the affine transformation matrix mapping at least one set of labeled point clouds to a set of standard point clouds. The corresponding target bounding box is then subjected to an affine transformation using this matrix to obtain the corrected bounding box. Affine transformation can accurately represent the mapping relationship between point cloud sets, and the affine transformation matrix can then be used to achieve accurate transformation of the target bounding box. This allows the mapping relationship between the labeled point cloud set and the standard point cloud set to be applied to the transformation of the target bounding box, improving the transformation accuracy and precision of the target bounding box.

[0105] As an example, using an affine transformation algorithm, the affine transformation matrix mapping at least one labeled point cloud set to a standard point cloud set is calculated, including:

[0106] By using a point cloud matching algorithm, the labeled point cloud set is paired with the standard point cloud set to obtain at least one point cloud mapping pair. The point cloud mapping pair includes the labeled point cloud data in the labeled point cloud set and the standard point cloud data in the standard point cloud set that has a mapping relationship with the labeled point cloud data.

[0107] By using an affine transformation algorithm, at least one set of point cloud mapping pairs is subjected to affine calculations to obtain the affine transformation matrix that maps the labeled point cloud set to the standard point cloud set.

[0108] Optionally, a point cloud matching algorithm is used to pair the labeled point cloud set with the standard point cloud set to obtain at least one set of point cloud mapping pairs. This can include: inputting the labeled point cloud set and the standard point cloud set into the point cloud matching algorithm, and pairing the point cloud data using the algorithm to obtain at least one set of point cloud mapping pairs. A point cloud mapping pair can include labeled point cloud data and standard point cloud data that have a mapping relationship.

[0109] The point cloud matching algorithm can be any type of point cloud matching algorithm, such as ICP (Iterative Closest Point) or NICP (Normal Iterative Closest Point). In this embodiment, the specific type of point cloud matching algorithm is not limited.

[0110] For ease of understanding, assume that the standard point cloud set uses P. predThis indicates that at least one set of labeled point clouds uses This indicates that k represents the number of at least one set of labeled point clouds or the number of at least one target bounding box. Label the target box The corresponding affine transformation matrix.

[0111] Therefore, by using a point cloud matching algorithm, the calculation is performed. Mapping to P pred Affine transformation matrix: T i (i = 1, ..., K), T i It can be represented as:

[0112]

[0113] Furthermore, the affine transformation matrix T mapping the labeled point cloud set to the standard point cloud set is obtained. i Then, the affine transformation calculation formula and the affine transformation matrix T can be used. i The target annotation box corresponding to the annotation point cloud set. Perform affine transformation calculations to obtain the target bounding box. Corresponding correction frame The formula for calculating radiometric transformation is:

[0114] The technical solution disclosed herein uses a point cloud matching algorithm to pair labeled point cloud sets with standard point cloud sets, achieving point-to-point pairing and obtaining at least one set of point cloud mapping pairs. Through affine calculations using these at least one set of point cloud mapping pairs, the affine transformation matrix mapping the labeled point cloud set to the standard point cloud set can be obtained. By pairing point cloud data, the mapping relationship between the labeled point cloud set and the standard point cloud set can be clearly defined, thereby enabling point-based computation, achieving smaller-dimensional affine calculations, and improving the accuracy of affine transformations.

[0115] Figure 5 This is a schematic diagram of the third embodiment of the present disclosure. The difference from the preceding embodiments is that, based on multiple bounding boxes in the training set, at least one target bounding box that matches the detection box is selected, including:

[0116] 501. Group the multiple bounding boxes in the training set to obtain at least one set of bounding boxes. Each set of bounding boxes is associated with at least one bounding box and a template bounding box. The template bounding box is a bounding box determined from at least one bounding box.

[0117] Optionally, step 501 may include: randomly dividing the multiple bounding boxes into at least one set of bounding boxes. Alternatively, step 501 may further include: dividing the multiple bounding boxes into at least one set of bounding boxes using a hash algorithm. Random partitioning or hash partitioning can achieve rapid partitioning of multiple bounding boxes.

[0118] A template annotation box can refer to the annotation box with the highest annotation quality determined from at least one annotation box.

[0119] 502. From at least one set of bounding boxes, determine a set of K target bounding boxes that match the detection boxes.

[0120] Optionally, step 502 may include: selecting a set of K target bounding boxes from at least one set of bounding boxes that satisfy the matching conditions of the detection boxes. The matching conditions may include the matching degree of the bounding box set being greater than or equal to a matching degree threshold, or the matching degree of the bounding box set belonging to the top K target matching degrees with the highest matching degrees.

[0121] For example, the matching degree between at least one set of bounding boxes and the detection box can be determined, and the set of target bounding boxes with a matching degree greater than or equal to the matching degree threshold can be determined to obtain a set of K target bounding boxes that match the detection box.

[0122] K is a positive integer greater than or equal to 1, and can be predetermined. For example, K can be set to 2.

[0123] 503. Determine that the template annotation box associated with each of the K target annotation box sets is at least one target annotation box.

[0124] Optionally, step 503 may include: sequentially reading the template annotation boxes associated with the K target annotation box sets according to the template annotation boxes corresponding to at least one set of annotation boxes.

[0125] In another possible design, in order to make the template annotation box more effective at least one target annotation box, step 503 may also include: determining whether the template annotation box of the target annotation box set meets the usage conditions; if so, then determining the template annotation box as the target annotation box.

[0126] Furthermore, determining whether the template annotation boxes in the target annotation box set meet the usage conditions may include: calculating the size ratio between the size of the template annotation boxes and the size of the detection boxes, and determining whether this size ratio is less than a ratio threshold. If so, the template annotation box is determined to meet the usage conditions; otherwise, it is determined that the template annotation box does not meet the usage conditions. If the size ratio between the template annotation boxes and the detection boxes differs significantly, it may lead to excessive transformation of the template annotation boxes relative to the detection boxes, resulting in negative effects. In this case, the template annotation box can be discarded, making the boxes participating in the final selection of target detection boxes more effective, reducing the participation of invalid template annotation boxes in the selection of target detection boxes, and improving selection efficiency and accuracy.

[0127] Furthermore, calculating the size ratio between the size of the template annotation box of the target annotation box set and the size of the detection box can include at least one of the following: the length ratio, width ratio, height ratio, and volume ratio between the template annotation box and the detection box of the target annotation box set.

[0128] In this disclosed technical solution, by grouping multiple bounding boxes, bounding boxes of the same type can be grouped into the same set, and the resulting set of at least one bounding box can represent a class of bounding boxes. Then, after determining K target bounding box sets that match the detection box from the at least one bounding box set, a set of K target bounding boxes matching the detection box can be obtained. Template bounding boxes associated with each of the K target bounding box sets matching the detection box are used as at least one target bounding box. Through classification and matching selection, at least one target bounding box can be quickly obtained, avoiding the dispersion of multiple bounding boxes in the detection box optimization process. By grouping bounding boxes of the same type through classification and selecting the corresponding template bounding boxes as target bounding boxes, the efficiency of obtaining at least one target bounding box is effectively improved.

[0129] As an optional implementation, determining a set of K target bounding boxes that match the detection boxes from at least one set of bounding boxes includes:

[0130] Based on at least one labeled box associated with the labeled box set, calculate the matching degree between the labeled box set and the detection box, and obtain the matching degree corresponding to at least one labeled box set respectively;

[0131] Based on the matching degree corresponding to at least one set of bounding boxes, determine the K sets of target bounding boxes with the highest matching degree.

[0132] The matching degree between the set of labeled boxes and the detection boxes can be obtained by calculating the distance between at least one labeled box associated with the set of labeled boxes and the detection box.

[0133] In this embodiment, calculating the matching degree between the set of labeled boxes and the corresponding detection boxes based on at least one labeled box associated with the set of labeled boxes may include: calculating the box distance between the set of labeled boxes and the corresponding detection boxes based on at least one labeled box associated with the set of labeled boxes, and using the box distance as the matching degree. The distance may include the box distance between at least one labeled box and the detection box.

[0134] The step of determining the set of K target bounding boxes with the highest matching degree based on the matching degree corresponding to at least one set of bounding boxes may include: sorting the matching degrees corresponding to at least one set of bounding boxes, and determining the set of K target bounding boxes with the highest matching degree from the sorted set of at least one set of bounding boxes.

[0135] Among them, the set of K target bounding boxes with the highest matching degree can refer to the set of bounding boxes associated with the top K matching degrees with the highest matching degree values ​​in at least one set of bounding boxes.

[0136] In the technical solution disclosed herein, based on at least one annotation box associated with a set of annotation boxes, the matching degree between the set of annotation boxes and the detection box can be calculated. Then, the matching process between at least one set of annotation boxes and the detection box is calculated in detail to obtain the matching degree corresponding to each of the at least one set of annotation boxes. By using the matching degree corresponding to each of the at least one set of annotation boxes, the selection of K target annotation box sets can be achieved directly, quickly and accurately, thereby improving the selection efficiency and accuracy of K target annotation box sets.

[0137] Furthermore, based on any of the above embodiments, the matching degree between the set of annotation boxes and the corresponding detection boxes is calculated according to at least one annotation box associated with the set of annotation boxes, including:

[0138] Based on at least one labeled box associated with the set of labeled boxes, calculate the box distance between the labeled box and the detection box to obtain the box distance corresponding to at least one labeled box respectively;

[0139] The average distance between the boxes corresponding to at least one labeled box is calculated, and the obtained average distance is used as the matching degree between the set of labeled boxes and the corresponding detection boxes.

[0140] Optionally, the frame distance can refer to the distance obtained by calculating the distance between at least one annotation frame and one detection frame. The steps for calculating the frame distance between the annotation frame and the detection frame may include: determining the position coordinates of the annotation frame and the detection frame, and calculating the frame distance between the template annotation frame and the detection frame based on the position coordinates of the annotation frame and the detection frame. The formula for calculating the coordinate distance can be found in the description of the following embodiments.

[0141] Alternatively, the calculation step of the distance between the bounding box and the detection box may further include: determining the feature vector of the bounding box and the feature vector of the detection box, and calculating the distance between the bounding box and the detection box based on the feature vectors of the bounding box and the detection box. The formula for calculating the vector distance can be found in the following embodiments.

[0142] In this embodiment, the average distance calculation for the distances of at least one labeled box can include: adding the distances of the boxes corresponding to at least one labeled box to obtain a sum of distances, determining the number of boxes of at least one labeled box, and calculating the quotient of the sum of distances and the number of boxes, which can be determined as the average distance.

[0143] In this disclosed technical solution, at least one bounding box associated with a set of bounding boxes is used to calculate the box distance between each bounding box and a detection box, ensuring that at least one bounding box participates in the calculation process. Then, the average of the box distances corresponding to each bounding box is calculated, and the resulting average distance is used as the matching degree between the bounding box and the detection box. This averaging method ensures that at least one bounding box in the bounding box set participates in the matching process with the detection box, resulting in a more balanced representation of the matching degree. Consequently, when selecting from a set of K target bounding boxes using the matching degrees corresponding to each of the at least one bounding box set, the selection is more stable and the selection accuracy is higher.

[0144] Furthermore, based on any of the above embodiments, the matching degree between the set of annotation boxes and the corresponding detection boxes is calculated according to at least one annotation box associated with the set of annotation boxes, including:

[0145] Determine the template annotation boxes corresponding to at least one set of annotation boxes;

[0146] Calculate the distance between the template standard boxes of at least one set of labeled boxes and the detection boxes;

[0147] The box distance of the annotation box set is determined as the matching degree between the annotation box set and the detection box, and the matching degree corresponding to at least one annotation box set is obtained.

[0148] Optionally, the template annotation box can be an annotation box determined from the annotation box set.

[0149] Optionally, the calculation steps for the frame distance between the template annotation box and the detection box may include: determining the position coordinates of the template annotation box and the detection box, and calculating the frame distance between the template annotation box and the detection box based on the position coordinates of the template annotation box and the detection box.

[0150] Furthermore, the coordinate distance between the position coordinates of the template annotation box and the position coordinates of the detection box can be calculated using the coordinate distance formula, and this coordinate distance can be determined as the box distance. The coordinate distance formula can be, for example: Where (x1, y1, z1) are the position coordinates of the template annotation box, and (x2, y2, z2) are the position coordinates of the detection box.

[0151] Optionally, the step of calculating the box distance between the template annotation box and the detection box may include: determining the feature vector of the template annotation box and the feature vector of the detection box, and calculating the box distance between the template annotation box and the detection box based on the feature vector of the template annotation box and the feature vector of the detection box.

[0152] Furthermore, the vector distance between the position coordinates of the template annotation box and the position coordinates of the detection box can be calculated according to the vector distance formula, and this vector distance can be determined as the box distance.

[0153] The vector distance formula can be, for example, the Euclidean distance formula, the Mahalanobis distance formula, the Manhattan distance formula, etc., and this embodiment does not impose too many restrictions on it.

[0154] In the technical solution disclosed herein, the distance between the template annotation boxes and the detection boxes in the annotation box set is calculated, and the obtained box distance can be used as the matching degree between the annotation box set and the detection boxes. The template annotation boxes are the annotation boxes that best represent the annotation box set. Using the box distance between the template annotation boxes and the detection boxes as the matching degree can make the matching degree more accurate and better reflect the matching effect between the annotation box set and the detection boxes.

[0155] As an optional implementation, after grouping multiple bounding boxes in the training set to obtain at least one set of bounding boxes, the method further includes:

[0156] Based on at least one annotation box corresponding to the annotation box set, extract the template annotation box corresponding to the annotation box set to obtain the template annotation box corresponding to at least one annotation box set respectively.

[0157] Optionally, extracting template annotation boxes corresponding to the annotation box set based on at least one annotation box corresponding to the annotation box set may include: randomly determining an annotation box as the template annotation box from at least one annotation box corresponding to the annotation box set.

[0158] Furthermore, extracting template annotation boxes corresponding to the annotation box set based on at least one annotation box corresponding to the annotation box set may include: selecting the annotation box with the highest quality evaluation from at least one annotation box corresponding to the annotation box set as the template annotation box.

[0159] In the technical solution disclosed herein, template annotation boxes corresponding to a set of annotation boxes can be extracted from at least one annotation box corresponding to a set of annotation boxes. This makes the template annotation boxes originate from the set of annotation boxes, better reflecting the commonalities of at least one annotation box within the set. The template annotation boxes can effectively represent the set of annotation boxes. Furthermore, by using the template annotation boxes associated with K sets of target annotation boxes as at least one target annotation box, more effective selection of target annotation boxes can be achieved with less computational complexity.

[0160] In one possible design, based on at least one label box corresponding to the label box set, the template label box corresponding to the label box set is extracted, including:

[0161] Perform quality evaluation on at least one annotation box corresponding to the set of annotation boxes to obtain quality evaluation data for at least one annotation box;

[0162] Based on the quality evaluation data corresponding to at least one annotation box, select the annotation box with the largest quality evaluation data as the template annotation box of the annotation box set.

[0163] Optionally, performing quality evaluation on at least one annotation box corresponding to the annotation box set to obtain quality evaluation data for at least one annotation box may include: displaying a rating page for at least one annotation box corresponding to the annotation box set, responding to a user's scoring operation triggered on the rating page for at least one annotation box, obtaining quality scores for at least one annotation box, and using the quality scores for at least one annotation box as quality evaluation data.

[0164] In this embodiment, selecting the label box with the largest quality evaluation data as the template label box of the label box set based on the quality evaluation data corresponding to at least one label box may include: sorting the quality evaluation data corresponding to at least one label box, and selecting the label box with the largest quality evaluation data from the sorted at least one label box as the template label box.

[0165] In the technical solution disclosed herein, during the process of extracting template annotation boxes from a set of annotation boxes, a quality evaluation method can be used to determine the annotation box with the highest quality evaluation data from at least one annotation box as the template annotation box, so that the template annotation box has the highest quality evaluation. Then, the annotation box with the highest quality evaluation is used as the template annotation box for the annotation box set, achieving accurate and effective acquisition of template annotation boxes for each annotation box set.

[0166] To achieve accurate and effective quality evaluation, furthermore, quality evaluation is performed on at least one annotation box corresponding to the annotation box set to obtain quality evaluation data corresponding to at least one annotation box, which may include:

[0167] Based on at least one bounding box corresponding to the bounding box set, determine the number of point clouds in the point cloud data surrounded by the corresponding training data for at least one bounding box;

[0168] The point cloud data of the corresponding training data of the bounding boxes are used as the quality evaluation data of the bounding boxes to obtain the quality evaluation data corresponding to at least one bounding box.

[0169] Optionally, determining the number of point clouds in the point cloud data surrounded by the corresponding training data based on at least one annotation box corresponding to the annotation box set may include: determining the coordinate range of the annotation box in the training data, and the point cloud coordinates corresponding to multiple point cloud data in the training data, determining the valid point cloud data whose point cloud coordinates are within the coordinate range of the annotation box, and determining the number of valid point cloud data, so as to obtain the number of point clouds in the point cloud data surrounded by the annotation box in the training data.

[0170] In this context, using the point cloud data of the bounding box in the corresponding training data as the quality evaluation data of the bounding box can mean determining the point cloud data of the bounding box in the corresponding training data as the quality evaluation data of the bounding box.

[0171] The number of point clouds can be defined as the number of point clouds within the corresponding training data bounded by the bounding boxes.

[0172] In the technical solution disclosed herein, the number of points in the point cloud data surrounded by the bounding box in the training data can be used as the quality evaluation data for the bounding box. The larger the number of points in the surrounding point cloud data, the higher the annotation effect of the bounding box on the object, and the higher the annotation quality of the bounding box. Therefore, using the number of points in the point cloud data as the quality evaluation data for the bounding box effectively improves the quality evaluation effect of the bounding box.

[0173] Furthermore, based on any of the above embodiments, multiple bounding boxes in the training set are grouped to obtain at least one set of bounding boxes, including:

[0174] Based on multiple bounding boxes in the training set, feature vectors of the bounding boxes are extracted to obtain feature vectors corresponding to each bounding box.

[0175] Based on the clustering algorithm, the feature vectors corresponding to multiple bounding boxes are clustered to obtain at least one set of bounding boxes.

[0176] Optionally, extracting the feature vector of the bounding box may include: extracting the feature vector of the bounding box based on its geometric information.

[0177] Furthermore, the geometric information of the annotation box can be represented as: b = [x, y, z, l, w, h, θ].

[0178] Where (x, y, z) are the position coordinates of the bounding box, l is the length of the bounding box, w is the width of the bounding box, and θ is the direction, such as the heading angle. Based on the geometric information of the bounding box, combined with vector transformation formulas, the geometric information can be converted into feature vectors to reduce the information dimensionality.

[0179] The vector transformation formula can be expressed as:

[0180] The clustering algorithm in this embodiment can be any clustering algorithm such as K-means clustering algorithm, and this embodiment does not impose too many restrictions on it.

[0181] In the technical solution disclosed herein, feature vectors of multiple bounding boxes in a training set can be extracted, and clustering calculations can be performed on the bounding boxes using these feature vectors to obtain at least one set of bounding boxes. By accurately representing the feature information of each bounding box and then using the feature vectors for clustering, fast and accurate grouping can be achieved.

[0182] Furthermore, based on any of the above embodiments, after verifying the detection box according to at least one target bounding box to obtain the target detection box, the method further includes:

[0183] Based on the 3D point cloud and the target detection box, generate the detection results of the target object;

[0184] Output the detection results.

[0185] Optionally, generating the detection result of the target object based on the 3D point cloud and the target detection box may include: rendering the target detection box in the 3D point cloud to obtain the detection page of the target object, and using the detection page as the detection result of the target object.

[0186] Taking the field of assisted driving as an example, outputting detection results can include: sending the detection results to the in-vehicle display screen and controlling the in-vehicle display screen to display the detection results.

[0187] In this embodiment, the detection result of the target object can be generated based on the 3D point cloud and the target detection box, and then output. By outputting the detection result, the target object detection interaction can be realized, improving the user experience.

[0188] To facilitate understanding of the technical solution of this disclosure, the application scenarios of the technical solution of this disclosure will be described in detail using vehicles as the target object.

[0189] refer to Figure 6 For the 3D point cloud 601, the detection box 602 surrounding the vehicle can be detected from the 3D point cloud 601.

[0190] Next, multiple bounding boxes that belong to the same vehicle category can be identified from the training set. These multiple bounding boxes can be represented as: Where n represents the nth label box in the label box set, and the value of n is 1-N, N is the number of label boxes in the set, and c is the vehicle category identifier.

[0191] According to the technical solution of this disclosure, multiple bounding boxes can be clustered to obtain at least one set of bounding boxes. Where M represents the number of template annotation boxes. Each set of annotation boxes is associated with at least one annotation box. For each set of annotation boxes, a corresponding template annotation box can be determined, thus obtaining the template annotation boxes corresponding to at least one set of annotation boxes, as detailed below. Figure 6 The template annotation box shown is 603.

[0192] Next, template filtering is performed. The specific filtering process may include: calculating the frame distance between at least one template bounding box and the detection box, and determining the frame distance of the bounding box set as the matching degree between the bounding box set and the detection box, thus obtaining the matching degree corresponding to at least one bounding box set. Then, using the matching degrees corresponding to at least one bounding box set, the K target bounding box sets with the highest matching degrees are determined, and these K target bounding box sets are used as at least one target bounding box. The template bounding boxes associated with each of the K target bounding box sets can be represented as follows: K is the number of at least one target bounding box, and j is a positive integer from 1 to K. Assume that the obtained target bounding boxes are as shown in Figure 604.

[0193] Subsequently, using point cloud matching and affine transformation algorithms, at least one target bounding box is subjected to an affine transformation to obtain corresponding correction boxes for each target bounding box. Finally, the target detection box 605 is determined from the at least one correction box and the predicted box. The specific correction process can be referred to the description in the foregoing embodiments, and will not be repeated here.

[0194] Figure 7 This is a schematic diagram of the fourth embodiment of the present disclosure, with reference to... Figure 7 The point cloud-based target detection device shown 700 may include the following units:

[0195] The acquisition unit 701 is used to acquire a detection box obtained from the 3D point cloud, wherein the detection box is a detection box that surrounds the target object in the 3D point cloud;

[0196] The filtering unit 702 is used to filter at least one target bounding box that matches the detection box based on multiple bounding boxes in the training set.

[0197] The correction unit 703 is used to perform verification processing on the detection box based on at least one target bounding box to obtain the target detection box.

[0198] As one embodiment, the correction unit includes:

[0199] A geometric correction module is used to transform at least one target bounding box into a correction box that matches the geometric information of the detection box, thereby obtaining at least one correction box;

[0200] The target determination module is used to determine the target detection box based on at least one calibration box and a detection box.

[0201] As another embodiment, the target determination module includes:

[0202] The candidate determination submodule is used to determine at least one candidate box based on at least one correction box and one detection box;

[0203] The quantity determination submodule is used to determine the number of point cloud data corresponding to at least one candidate box in the 3D point cloud. The number of point cloud data is the number of point cloud data contained in the candidate box in the 3D point cloud.

[0204] The target determination submodule is used to determine the candidate box corresponding to the maximum number of point clouds as the target detection box based on the number of point clouds corresponding to at least one candidate box.

[0205] As another embodiment, the geometric correction module includes:

[0206] The standard extraction submodule is used to enlarge the detection box by N times according to the original aspect ratio and width-to-height ratio to obtain the standard box, and extract the standard point cloud set that belongs to the standard box from the 3D point cloud;

[0207] The annotation extraction submodule is used to extract the annotation point cloud set corresponding to at least one target annotation box based on the training data corresponding to at least one target annotation box. The training data is the training 3D point cloud that participated in the training, and the training data is associated with the annotation boxes.

[0208] The annotation correction submodule is used to transform at least one target annotation box into a correction box based on the annotation point cloud set corresponding to each of the at least one target annotation boxes and the standard point cloud set, thereby obtaining at least one correction box.

[0209] As another embodiment, the annotation and correction submodule is specifically used for:

[0210] Using an affine transformation algorithm, calculate the affine transformation matrix that maps at least one set of labeled point clouds to a set of standard point clouds.

[0211] Affine transformation is performed on the corresponding target bounding boxes based on the affine transformation matrix of the labeled point cloud set to obtain at least one correction box corresponding to each target bounding box.

[0212] As another embodiment, the annotation and correction submodule is specifically used for:

[0213] By using a point cloud matching algorithm, the labeled point cloud set is paired with the standard point cloud set to obtain at least one point cloud mapping pair. The point cloud mapping pair includes the labeled point cloud data in the labeled point cloud set and the standard point cloud data in the standard point cloud set that has a mapping relationship with the labeled point cloud data.

[0214] By using an affine transformation algorithm, at least one set of point cloud mapping pairs is subjected to affine calculations to obtain the affine transformation matrix that maps the labeled point cloud set to the standard point cloud set.

[0215] As another embodiment, the screening unit includes:

[0216] The annotation grouping module is used to group multiple annotation boxes in the training set to obtain at least one set of annotation boxes. Each set of annotation boxes is associated with at least one annotation box and a template annotation box. The template annotation box is an annotation box determined from at least one annotation box.

[0217] The target matching module is used to determine a set of K target bounding boxes that match the detection boxes from at least one set of bounding boxes;

[0218] The target determination module, which is associated with the template annotation box of each of the K target annotation box sets, has at least one target annotation box.

[0219] As yet another embodiment, the target matching module includes:

[0220] The first calculation submodule is used to calculate the matching degree between the set of annotation boxes and the detection box based on at least one annotation box associated with the set of annotation boxes, and to obtain the matching degree corresponding to at least one set of annotation boxes respectively;

[0221] The first matching submodule is used to determine the K target bounding box sets with the highest matching degree based on the matching degree corresponding to at least one set of bounding boxes.

[0222] As yet another embodiment, the first calculation submodule is specifically used for:

[0223] Based on at least one labeled box associated with the set of labeled boxes, calculate the box distance between the labeled box and the detection box to obtain the box distance corresponding to at least one labeled box respectively;

[0224] The average distance between the boxes corresponding to at least one labeled box is calculated, and the obtained average distance is used as the matching degree between the set of labeled boxes and the corresponding detection boxes.

[0225] As yet another embodiment, the first calculation submodule is specifically used for:

[0226] Determine the template annotation boxes corresponding to at least one set of annotation boxes;

[0227] Calculate the distance between the template standard boxes of at least one set of labeled boxes and the detection boxes;

[0228] The box distance of the annotation box set is determined as the matching degree between the annotation box set and the detection box, and the matching degree corresponding to at least one annotation box set is obtained.

[0229] As yet another embodiment, it also includes:

[0230] The template acquisition unit is used to extract template annotation boxes corresponding to the annotation box set based on at least one annotation box corresponding to the annotation box set, and obtain template annotation boxes corresponding to at least one annotation box set respectively.

[0231] As another embodiment, the template acquisition unit includes:

[0232] Perform quality evaluation on at least one annotation box corresponding to the set of annotation boxes to obtain quality evaluation data for at least one annotation box;

[0233] The quality selection module is used to select the label box with the largest quality evaluation data as the template label box of the label box set based on the quality evaluation data corresponding to at least one label box.

[0234] As yet another embodiment, the quality selection module includes:

[0235] The quantity determination submodule is used to determine the number of point clouds in the point cloud data surrounded by the corresponding training data based on the at least one annotation box corresponding to the annotation box set.

[0236] The quality screening submodule is used to take the number of points in the corresponding training data of the bounding box as the quality evaluation data of the bounding box, and obtain the quality evaluation data corresponding to at least one bounding box.

[0237] As another embodiment, the annotation grouping module includes:

[0238] The vector extraction submodule is used to extract the feature vectors of multiple bounding boxes based on the training set, and obtain the feature vectors corresponding to each bounding box.

[0239] The labeling and clustering submodule is used to cluster the feature vectors corresponding to multiple label boxes according to the clustering algorithm to obtain at least one set of label boxes.

[0240] As yet another embodiment, it also includes:

[0241] The generation unit is used to generate the detection results of the target object based on the 3D point cloud and the target detection box;

[0242] The output unit is used to output the detection results.

[0243] It should be noted that the target object in this embodiment is not specific to any particular user and does not reflect the personal information of any specific user. It should also be noted that the training set in this embodiment comes from a publicly available dataset.

[0244] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0245] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0246] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0247] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0248] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0249] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0250] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as point cloud-based object detection methods. For example, in some embodiments, the point cloud-based object detection method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the point cloud-based object detection method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform point cloud-based object detection methods by any other suitable means (e.g., by means of firmware).

[0251] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0252] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0253] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0254] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0255] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0256] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0257] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0258] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A point cloud-based target detection method, comprising: Obtain a detection box from a 3D point cloud, wherein the detection box is a detection box that surrounds the target object in the 3D point cloud; Based on multiple bounding boxes in the training set, feature vectors of the bounding boxes are extracted, and the feature vectors are clustered into at least one set of bounding boxes according to a clustering algorithm. At least one target bounding box that matches the detection box is selected from the at least one set of bounding boxes. The bounding boxes are the true bounding boxes that enclose objects in the training data; The detection frame is enlarged by N times according to its original aspect ratio and width-to-height ratio to obtain a standard frame. The standard point cloud set belonging to the standard frame is extracted from the 3D point cloud. The size of the standard frame is a preset multiple of the size of the detection frame. Based on the training data corresponding to at least one of the target bounding boxes, extract at least one set of labeled point clouds corresponding to each of the target bounding boxes, wherein the training data is the training 3D point cloud participating in the training, and the training data is associated with the bounding boxes; Based on the point cloud set corresponding to at least one of the target bounding boxes, and combined with the standard point cloud set, at least one of the target bounding boxes is transformed into correction boxes to obtain at least one correction box; The target detection box is determined based on at least one of the correction boxes and the detection boxes; Based on the target detection box, position tracking processing is performed on the target object to obtain the motion trajectory of the target object, and the motion trajectory of the target object is output.

2. The method according to claim 1, wherein, Determining the target detection box based on at least one of the correction boxes and the detection boxes includes: Based on at least one of the correction boxes and the detection boxes, at least one candidate box is determined; Determine the number of point clouds corresponding to at least one of the candidate boxes in the 3D point cloud, wherein the number of point clouds is the number of point cloud data contained in the 3D point cloud by the candidate box; Based on the number of point clouds corresponding to at least one of the candidate boxes, the candidate box corresponding to the largest number of point clouds is determined as the target detection box.

3. The method according to claim 1, wherein, The step of transforming at least one target bounding box into a correction box based on the point cloud set corresponding to each of the at least one target bounding box, and combining it with the standard point cloud set, to obtain at least one correction box includes: Using an affine transformation algorithm, calculate at least one affine transformation matrix mapping the labeled point cloud set to the standard point cloud set; Based on the affine transformation matrix of the labeled point cloud set, perform an affine transformation on the corresponding target labeled boxes to obtain at least one correction box corresponding to each of the target labeled boxes.

4. The method according to claim 3, wherein, The step of calculating at least one affine transformation matrix mapping the labeled point cloud set to the standard point cloud set using an affine transformation algorithm includes: By using a point cloud matching algorithm, the labeled point cloud set is paired with the standard point cloud set to obtain at least one point cloud mapping pair. The point cloud mapping pair includes the labeled point cloud data in the labeled point cloud set and the standard point cloud data in the standard point cloud set that has a mapping relationship with the labeled point cloud data. The affine transformation algorithm is used to perform affine calculations on at least one set of point cloud mapping pairs to obtain the affine transformation matrix that maps the labeled point cloud set to the standard point cloud set.

5. The method according to any one of claims 1-4, wherein, Filtering at least one target bounding box from the at least one set of bounding boxes that matches the detection box includes: From at least one set of labeled boxes, determine a set of K target labeled boxes that match the detection box; each set of labeled boxes is associated with at least one labeled box and a template labeled box, wherein the template labeled box is a labeled box determined from at least one labeled box; The template annotation box associated with each of the K target annotation box sets is determined to be at least one of the target annotation boxes.

6. The method according to claim 5, wherein, The step of determining a set of K target bounding boxes that match the detection box from at least one set of bounding boxes includes: Based on at least one of the labeled boxes associated with the labeled box set, calculate the matching degree between the labeled box set and the detection box to obtain the matching degree corresponding to at least one of the labeled box sets respectively; Based on the matching degree corresponding to at least one of the aforementioned bounding box sets, determine the K target bounding box sets with the highest matching degree.

7. The method according to claim 6, wherein, The step of calculating the matching degree between the set of labeled boxes and the detection box based on at least one labeled box associated with the set of labeled boxes includes: Based on at least one of the labeled boxes associated with the set of labeled boxes, the box distance between the labeled box and the detection box is calculated to obtain the box distance corresponding to at least one of the labeled boxes respectively; The average distance between the boxes corresponding to at least one of the labeled boxes is calculated, and the obtained average distance is used as the matching degree between the set of labeled boxes and the corresponding detection boxes.

8. The method according to claim 6, wherein, The step of calculating the matching degree between the set of labeled boxes and the detection box based on at least one labeled box associated with the set of labeled boxes includes: Determine at least one template annotation box corresponding to each of the aforementioned annotation box sets; Calculate the frame distance between the template annotation boxes of at least one of the annotation box sets and the detection boxes; The box distance of the set of labeled boxes is determined as the matching degree between the set of labeled boxes and the detection box, and the matching degree corresponding to at least one set of labeled boxes is obtained.

9. The method according to claim 5, wherein, After clustering the feature vectors into at least one set of bounding boxes using a clustering algorithm, the method further includes: Based on at least one of the annotation boxes corresponding to the annotation box set, extract the template annotation box corresponding to the annotation box set to obtain at least one template annotation box corresponding to each of the annotation box sets.

10. The method according to claim 9, wherein, The step of extracting template annotation boxes corresponding to the annotation box set based on at least one annotation box corresponding to the annotation box set includes: Perform quality evaluation on at least one of the annotation boxes corresponding to the set of annotation boxes to obtain quality evaluation data corresponding to at least one of the annotation boxes respectively; Based on the quality evaluation data corresponding to at least one of the labeled boxes, the labeled box with the largest quality evaluation data is selected as the template labeled box of the labeled box set.

11. The method according to claim 10, wherein, The step of performing quality evaluation on at least one of the annotation boxes corresponding to the set of annotation boxes to obtain quality evaluation data corresponding to at least one of the annotation boxes includes: Based on at least one of the annotation boxes corresponding to the set of annotation boxes, determine the number of point clouds in the point cloud data surrounded by the corresponding training data for at least one of the annotation boxes; The number of point clouds in the corresponding training data of the labeled boxes is used as the quality evaluation data of the labeled boxes, and at least one quality evaluation data corresponding to each labeled box is obtained.

12. The method according to any one of claims 1-4, wherein, After determining the target detection box based on at least one of the correction boxes and the detection boxes, the method further includes: Based on the 3D point cloud and the target detection box, the detection result of the target object is generated; Output the detection results.

13. A target detection device based on point clouds, comprising: An acquisition unit is used to acquire a detection box obtained from a 3D point cloud, wherein the detection box is a detection box that surrounds a target object in the 3D point cloud; The filtering unit is used to extract feature vectors of the multiple bounding boxes in the training set, cluster the feature vectors into at least one set of bounding boxes according to a clustering algorithm, and filter at least one target bounding box that matches the detection box from the at least one set of bounding boxes. The bounding boxes are the true bounding boxes that enclose objects in the training data; A correction unit is used to perform verification processing on the detection box based on at least one of the target bounding boxes to obtain the target detection box; The processing unit is used to perform position tracking processing on the target object based on the target detection box, obtain the motion trajectory of the target object, and output the motion trajectory of the target object; The correction unit includes: A geometric correction module is used to transform at least one of the target bounding boxes into corrected boxes that match the geometric information of the detection boxes, thereby obtaining at least one corrected box; A target determination module is configured to determine the target detection box based on at least one of the correction boxes and the detection boxes; The geometric correction module includes: The standard extraction submodule is used to enlarge the detection box by N times according to the original aspect ratio and width-to-height ratio to obtain a standard box, and extract the standard point cloud set belonging to the standard box from the 3D point cloud; The annotation extraction submodule is used to extract at least one set of annotation point clouds corresponding to at least one target annotation box based on the training data corresponding to at least one target annotation box, wherein the training data is the training 3D point cloud participating in the training, and the training data is associated with the annotation boxes. The annotation correction submodule is used to transform at least one of the target annotation boxes into correction boxes based on the annotation point cloud set corresponding to each of the at least one target annotation box and the standard point cloud set, thereby obtaining at least one correction box.

14. The apparatus according to claim 13, wherein, The target determination module includes: A candidate determination submodule is used to determine at least one candidate box based on at least one of the correction boxes and the detection boxes; The quantity determination submodule is used to determine the number of point clouds corresponding to at least one of the candidate boxes in the 3D point cloud, wherein the number of point clouds is the number of point cloud data contained in the 3D point cloud by the candidate box. The target determination submodule is used to determine the candidate box corresponding to the maximum number of point clouds as the target detection box based on the number of point clouds corresponding to at least one of the candidate boxes.

15. The apparatus according to claim 13, wherein, The annotation correction submodule is specifically used for: Using an affine transformation algorithm, calculate at least one affine transformation matrix mapping the labeled point cloud set to the standard point cloud set; Based on the affine transformation matrix of the labeled point cloud set, perform an affine transformation on the corresponding target labeled boxes to obtain at least one correction box corresponding to each of the target labeled boxes.

16. The apparatus according to claim 15, wherein, The annotation correction submodule is specifically used for: By using a point cloud matching algorithm, the labeled point cloud set is paired with the standard point cloud set to obtain at least one point cloud mapping pair. The point cloud mapping pair includes the labeled point cloud data in the labeled point cloud set and the standard point cloud data in the standard point cloud set that has a mapping relationship with the labeled point cloud data. The affine transformation algorithm is used to perform affine calculations on at least one set of point cloud mapping pairs to obtain the affine transformation matrix that maps the labeled point cloud set to the standard point cloud set.

17. The apparatus according to any one of claims 13-16, wherein, The filtering unit includes: The target matching module is used to determine a set of K target bounding boxes that match the detection box from at least one set of bounding boxes; the set of bounding boxes is associated with at least one bounding box and a template bounding box, wherein the template bounding box is a bounding box determined from at least one bounding box; The target determination module determines that at least one of the target annotation boxes is associated with the template annotation boxes of the K target annotation box sets.

18. The apparatus according to claim 17, wherein, The target matching module includes: The first calculation submodule is used to calculate the matching degree between the annotation box set and the detection box based on at least one of the annotation boxes associated with the annotation box set, and obtain the matching degree corresponding to at least one of the annotation box sets respectively; The first matching submodule is used to determine the K target bounding box sets with the highest matching degree based on the matching degree corresponding to at least one of the bounding box sets.

19. The apparatus according to claim 18, wherein, The first calculation submodule is specifically used for: Based on at least one of the labeled boxes associated with the set of labeled boxes, the box distance between the labeled box and the detection box is calculated to obtain the box distance corresponding to at least one of the labeled boxes respectively; The average distance between the boxes corresponding to at least one of the labeled boxes is calculated, and the obtained average distance is used as the matching degree between the set of labeled boxes and the corresponding detection boxes.

20. The apparatus according to claim 18, wherein, The first calculation submodule is specifically used for: Determine at least one template annotation box corresponding to each of the aforementioned annotation box sets; Calculate the frame distance between the template annotation boxes of at least one of the annotation box sets and the detection boxes; The box distance of the set of labeled boxes is determined as the matching degree between the set of labeled boxes and the detection box, and the matching degree corresponding to at least one set of labeled boxes is obtained.

21. The apparatus of claim 17, further comprising: The template acquisition unit is used to extract template annotation boxes corresponding to the annotation box set based on at least one annotation box corresponding to the annotation box set, and obtain template annotation boxes corresponding to at least one annotation box set respectively.

22. The apparatus according to claim 21, wherein, The template acquisition unit includes: The quality selection module is used to evaluate the quality of at least one of the annotation boxes corresponding to the set of annotation boxes, and obtain the quality evaluation data corresponding to at least one of the annotation boxes respectively. Based on the quality evaluation data corresponding to at least one of the labeled boxes, the labeled box with the largest quality evaluation data is selected as the template labeled box of the labeled box set.

23. The apparatus according to claim 22, wherein, The quality selection module includes: The quantity determination submodule is used to determine the number of point clouds in the point cloud data surrounded by the corresponding training data based on at least one of the annotation boxes corresponding to the annotation box set. The quality screening submodule is used to take the number of points in the corresponding training data of the labeled box as the quality evaluation data of the labeled box, and obtain the quality evaluation data corresponding to at least one labeled box.

24. The apparatus according to any one of claims 13-16, further comprising: The generation unit is used to generate the detection result of the target object based on the 3D point cloud and the target detection box; The output unit is used to output the detection results.

25. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

27. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-12.

Images