A method and system for shape estimation of dense and cluttered objects

Abstract

This invention provides a shape estimation method and system for dense and cluttered objects, belonging to the field of computer vision technology. The method includes: acquiring instance point clouds of each object to be evaluated, and acquiring template point clouds corresponding to the category of each object; inputting the instance point clouds and template point clouds of each object to be evaluated into a shape restoration network to obtain shape restoration parameters for each object output by the shape restoration network; the shape restoration network is trained using instance point cloud samples and corresponding real point cloud labels; and transforming the template point cloud corresponding to each object to be evaluated based on the shape restoration parameters to obtain a predicted point cloud for each object. This invention achieves high-precision shape estimation of objects to be evaluated in dense and cluttered scenes by combining observation information from instance point clouds with the category geometric prior of template point clouds and dynamically generating shape restoration parameters using a shape restoration network.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for shape estimation of dense and cluttered objects. Background Technology

[0002] With the development of automation and intelligent technologies, robots are being used more and more widely in dense and complex environments, such as logistics sorting, industrial assembly, and warehouse management. In these scenarios, robots need to accurately identify and manipulate objects that are occluded, overlapping, or partially visible in complex environments. The key to achieving this goal lies in accurately estimating the three-dimensional shape of the target object.

[0003] However, traditional techniques typically rely on single-view image data or directly utilize point cloud data acquired from 3D sensors for processing. In single-view methods, due to limited viewpoints and occlusion issues, the acquired point cloud data is often incomplete, severely limiting the accuracy of shape estimation. While multi-view point cloud fusion methods can partially alleviate the information loss problem, their performance remains unsatisfactory when handling multi-object scenes due to dense occlusion between objects and difficulties in point cloud separation. Furthermore, some techniques attempt to directly match target objects using predefined template shapes, but this method lacks the ability to dynamically model instantiated features and geometric details, resulting in poor generalization and difficulty in adapting to diverse needs involving unseen objects or categories.

[0004] Therefore, how to achieve complete and high-precision shape estimation of the object to be evaluated in dense and cluttered scenes has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides a method, system, electronic device, and storage medium for shape estimation of dense and cluttered objects, in order to overcome the deficiencies in the prior art and achieve complete and high-precision shape estimation of some observed objects to be evaluated in dense and cluttered scenes.

[0006] This invention provides a shape estimation method for densely packed and cluttered objects, comprising the following steps:

[0007] Instance point clouds of each object to be evaluated are obtained, and template point clouds corresponding to the category of each object to be evaluated are obtained; the instance point clouds are used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point clouds are used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated.

[0008] The instance point cloud and template point cloud of each object to be evaluated are input into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels.

[0009] The template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0010] According to the shape estimation method for dense and cluttered objects provided by the present invention, the step of obtaining the instance point cloud of each object to be evaluated specifically includes:

[0011] Obtain color images of a pile of objects consisting of multiple objects to be evaluated from multiple viewpoints, and obtain a depth image corresponding to each color image;

[0012] Each of the color images is input into the vocabulary segmentation model to obtain the instance segmentation mask and predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images from multiple perspectives;

[0013] The instance segmentation mask of each object to be evaluated under each viewpoint is combined with the corresponding depth image to obtain the initial point cloud of each object to be evaluated under each viewpoint.

[0014] Multiple rounds of iterative merging are performed on initial point clouds with the same prediction category from different viewpoints to obtain instance point clouds of each object to be evaluated.

[0015] According to the shape estimation method for dense and cluttered objects provided by the present invention, the step of iteratively merging initial point clouds with the same prediction category from different viewpoints to obtain an instance point cloud for each object to be evaluated specifically includes:

[0016] In each iteration, the first chamfer distance between the initial point with the same predicted category under each different viewpoint and the currently merged instance point cloud is calculated;

[0017] When it is determined that the first chamfer distance is less than a preset threshold, the initial point cloud is merged with the currently merged instance point cloud;

[0018] Repeat the above merging process and update the currently merged instance point cloud after each round of merging;

[0019] When it is determined that no new initial point cloud satisfies the condition that the first chamfer distance is less than a preset threshold, the iteration ends, and the final instance point cloud of each object to be evaluated is obtained.

[0020] According to the present invention, a shape estimation method for dense and cluttered objects is provided, wherein the shape restoration network includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model; the step of inputting the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network specifically includes:

[0021] The instance point cloud of each object to be evaluated is input into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic-level feature extraction on the input point cloud data.

[0022] The template point cloud of each object to be evaluated is input into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on the hierarchical or regional division of point cloud data.

[0023] By using a cross-attention mechanism, the fusion features of each object to be evaluated are extracted from the instance point cloud and template point cloud of each object to be evaluated.

[0024] The template features, fusion features, and instance features of each object to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network.

[0025] According to the present invention, a shape estimation method for dense and cluttered objects is provided, wherein the shape restoration parameters include a scaling factor and a point-by-point offset; the step of transforming the template point cloud corresponding to each object to be evaluated based on the shape restoration parameters of each object to be evaluated to obtain a predicted point cloud for each object to be evaluated specifically includes:

[0026] Multiply the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated.

[0027] The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0028] The present invention provides a shape estimation method for dense and cluttered objects, which further includes a training method for the shape recovery network:

[0029] Obtain instance point cloud samples, template point cloud samples, and real point cloud labels;

[0030] The instance point cloud sample and the template point cloud sample are input into the initial network to obtain the point-by-point offset sample and scaling factor sample output by the initial network.

[0031] The template point cloud sample is transformed based on the point-by-point offset sample and the scaling factor sample to obtain the predicted point cloud sample.

[0032] Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label;

[0033] Based on the second chamfer distance and the point-by-point offset samples, a loss function is constructed;

[0034] The initial network is trained with the goal of minimizing the loss function to obtain the shape recovery network.

[0035] According to the present invention, a shape estimation method for dense and cluttered objects is provided, wherein constructing a loss function based on the second chamfer distance and the point-by-point offset samples specifically includes:

[0036] The loss function is expressed by the following formula:

[0037] ;

[0038] In the formula, Let the loss function be... The actual point cloud labels, For the predicted point cloud sample, This is the distance of the second chamfer. For the point-by-point offset sample, It is the L2 norm. The sum of the L2 norms of the point-by-point offset samples is given, where n is the number of points in the predicted point cloud sample. This is a hyperparameter.

[0039] The present invention also provides a shape estimation system for dense and cluttered objects, comprising the following modules:

[0040] The acquisition module is used to acquire instance point clouds of each object to be evaluated and to acquire template point clouds corresponding to the category of each object to be evaluated; the instance point clouds are used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point clouds are used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated.

[0041] The processing module is used to input the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels.

[0042] The transformation module is used to transform the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the shape estimation method for dense and cluttered objects as described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the shape estimation method for dense, cluttered objects as described above.

[0045] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the shape estimation method for dense and cluttered objects as described above.

[0046] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0047] By acquiring instance point clouds and corresponding category template point clouds for each object to be evaluated, a foundation combining observation and prior knowledge is provided for shape estimation. Instance point clouds reflect the 3D point cloud data of the object under actual observation conditions, capturing its actual geometric distribution; however, they may be incomplete due to occlusion and data gaps in dense and cluttered environments. Template point clouds, serving as geometric priors for the categories, are generated through optimization based on diverse samples, providing a reliable reference benchmark for shape completion and compensating for insufficient observation in instance point clouds. The instance and template point clouds of each object to be evaluated are input into a shape restoration network, enabling the network to fuse features using the observational information of the instance point clouds and the prior information of the template point clouds, outputting shape restoration parameters for each object. These parameters include the scaling factor and point-by-point offset of the template point clouds, used for global adjustment and local correction of the template point cloud shape, thus ensuring consistency between the template point cloud and the actual observed geometric characteristics of the instance point clouds. In this way, the shape restoration network can utilize both the global geometric features of the template point clouds and dynamically adapt to the local observational characteristics of the instance point clouds, thereby achieving accurate estimation of object shapes in complex scenes. The shape restoration network is trained using instance point cloud samples and real point cloud labels to ensure that the network can learn the geometric mapping relationship between instance point clouds and template point clouds. Simultaneously, its generation capability is optimized by minimizing the geometric error between the predicted and real point clouds. By constructing a comprehensive loss function that includes chamfer distance and point-by-point offset regularization terms, the training process constrains local adjustments while ensuring global geometric consistency, further improving the prediction accuracy of shape restoration parameters. The template point cloud is transformed based on the shape restoration parameters. First, the global shape of the template point cloud is adjusted by a scaling factor to make its size and proportion closer to the observed characteristics of the instance point cloud. Then, the template point cloud is locally corrected using point-by-point offsets to refine its detailed shape, ultimately generating the predicted point cloud. The predicted point cloud retains the categorical geometric features of the template point cloud while fully adapting to the observed data of the instance point cloud, achieving complete and high-precision shape estimation of the object to be evaluated in dense and cluttered scenes. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is one of the flowcharts of the shape estimation method for dense and cluttered objects provided by the present invention.

[0050] Figure 2This is the second flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0051] Figure 3 This is a schematic diagram of the shooting scene of the multi-view RGB-D camera provided by the present invention.

[0052] Figure 4 This is the third flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0053] Figure 5 This is the fourth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0054] Figure 6 This is the fifth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0055] Figure 7 This is the sixth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0056] Figure 8 This is a schematic diagram of the shape estimation system for dense and messy objects provided by the present invention.

[0057] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0059] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0060] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0061] The following is combined with Figures 1-9 This invention describes the shape estimation method, system, electronic device, and storage medium for densely packed, cluttered objects.

[0062] Figure 1 This is one of the flowcharts illustrating the shape estimation method for densely packed, cluttered objects provided by this invention, such as... Figure 1 As shown, including but not limited to the following steps:

[0063] Step 101: Obtain the instance point cloud for each object to be evaluated, and obtain the template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated.

[0064] In dense and cluttered scenes, due to object occlusion, overlap, and scene complexity, single-view point clouds often lack sufficient information, making it difficult to accurately extract complete point cloud data for individual objects. Therefore, in step 101, by acquiring the instance point cloud of each object to be evaluated and combining it with the template point cloud corresponding to the category of each object, a complete and accurate geometric benchmark is provided for subsequent shape completion. In this process, by acquiring data from multiple perspectives, using a vocabulary segmentation model for object segmentation and category prediction, and combining a multi-round iterative merging method, the integrity and accuracy of the instance point clouds are ensured.

[0065] The following examples will detail the methods for obtaining the instance point cloud of each object to be evaluated and the template point cloud corresponding to the category of each object to be evaluated.

[0066] First, the specific steps for obtaining the instance point cloud of each object to be evaluated are explained in detail.

[0067] In one possible implementation, refer to Figure 2 , Figure 2 This is the second flowchart of the shape estimation method for dense and cluttered objects provided by the present invention, as shown below. Figure 2 As shown, in step 101, the method for obtaining the instance point cloud of each object to be evaluated specifically includes steps 201-204:

[0068] Step 201: Obtain color images of the object pile consisting of multiple objects to be evaluated from multiple viewpoints, and obtain the depth image corresponding to each color image.

[0069] Specifically, a multi-view RGB-D camera is used to photograph the pile of objects, capturing color images and corresponding depth images from each viewpoint. A diagram illustrating the specific shooting scene can be found here. Figure 3 , Figure 3 This is a schematic diagram of the shooting scene of the multi-view RGB-D camera provided by the present invention.

[0070] Color images record the texture and color features of the scene, which are crucial for subsequent object segmentation and category prediction using vocabulary segmentation models. Simultaneously, depth images record the spatial depth information of the object surfaces, providing the 2D and 3D data foundation required for point cloud generation, together with the color images. During data acquisition, ensuring the camera's position, angle, and viewpoint cover the main areas of the object cluster minimizes occlusion and blind spots, thereby improving the completeness and quality of the acquired point cloud data.

[0071] Step 202: Input each color image into the vocabulary segmentation model to obtain the instance segmentation mask and predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images from multiple perspectives;

[0072] Specifically, the vocabulary segmentation model is fine-tuned based on sample color images from multiple perspectives. With the semantic support of open vocabulary, this model can identify and segment unseen object categories.

[0073] To achieve accurate object segmentation and category prediction in dense and cluttered scenes, this solution employs the Detic open-vocabulary segmentation model. Detic is a segmentation model pre-trained on the large-scale ImageNet-21K dataset, possessing rich external knowledge and strong generalization capabilities. With the support of an open vocabulary, it can identify multiple categories in complex scenes, and even for unfamiliar object categories, it can effectively segment and classify them based on their semantic features.

[0074] To further improve the segmentation accuracy of the Detic model in specific scenarios, this approach fine-tunes it on a specific training set. The fine-tuning process incorporates an adapter layer. By adding a lightweight adapter module to the model, the resulting word segmentation model, after fine-tuning, can better adapt to unique visual features in dense and cluttered scenes, such as complex occlusions and fine-grained differences between similar categories. As part of the model, the adapter layer efficiently adjusts the output of the word segmentation model, making its segmentation performance more suitable for the task requirements of this approach.

[0075] After each acquired color image is input into the vocabulary segmentation model, the model performs semantic segmentation on the objects in each image, generating an instance segmentation mask and a corresponding category prediction label. The instance segmentation mask is a two-dimensional pixel-level segmentation result for each object, used to represent the region of each object in the image; the category prediction label provides semantic category information of the object. This combined segmentation and classification output can not only effectively handle complex occlusion and overlap problems in dense scenes, but also provide crucial semantic constraints for subsequent 3D point cloud generation.

[0076] Step 203: Combine the instance segmentation mask of each object to be evaluated in each view with the corresponding depth image to obtain the initial point cloud of each object to be evaluated in each view.

[0077] Specifically, from each viewpoint, the instance segmentation mask generated in step 202 is first pixel-aligned with the depth map corresponding to the color image. Each segmentation mask's region corresponds to the location range of an object in the 2D image, while the depth map provides depth information in 3D space for that location. By utilizing the camera's intrinsic and extrinsic parameters, the pixel coordinates of each segmentation mask are combined with the depth value to convert them into 3D spatial coordinates, ultimately generating the initial point cloud for each object to be evaluated from the current viewpoint. This process relies on the camera's projection relationship formula, calculating the object's position in 3D space using pixel coordinates and depth values.

[0078] This method accurately extracts the initial point cloud of each object from every viewpoint, reflecting its local geometric characteristics in 3D space. Notably, because the segmentation mask maps only the segmented region, interference from background or other object information is avoided, thus ensuring the purity and specificity of the generated point cloud. Furthermore, the initial point clouds from multiple views are collected from different angles, providing multi-view support for subsequent point cloud merging and effectively mitigating the problem of incomplete reconstruction caused by missing information from a single viewpoint.

[0079] Step 204: Perform multiple rounds of iterative merging on the initial point clouds with the same prediction category from different viewpoints to obtain the instance point cloud of each object to be evaluated.

[0080] In dense and cluttered scenes, since the initial point cloud under each view can only reflect the local geometric information of the object, and the point cloud fragments under different views often overlap or are inconsistent, in order to obtain the complete instance point cloud of each object to be evaluated, step 204 integrates the multi-view information into a unified point cloud representation by performing multiple rounds of iterative merging on the initial point clouds with the same prediction category under different views, thereby solving the problem of missing shape information caused by local occlusion and single-view limitation.

[0081] In one possible implementation, refer to Figure 4 , Figure 4 This is the third flowchart of the shape estimation method for dense and cluttered objects provided by the present invention, as shown below. Figure 4 As shown, step 204 specifically includes steps 301-304:

[0082] Step 301: In each iteration, calculate the first chamfer distance between the initial point with the same predicted category under each different viewpoint and the currently merged instance point cloud.

[0083] In dense and cluttered scenes, the initial point cloud often originates from multiple perspectives. Point cloud fragments from different perspectives may exhibit positional offsets or geometric differences. To ensure that only point clouds with consistent geometric structures are integrated during multiple iterations of merging, step 301 calculates the first chamfer distance between the initial point clouds with the same prediction category from different perspectives and the currently merged instance point cloud, using this distance as a basis for judging geometric consistency. The purpose of this step is to utilize the geometric properties of the chamfer distance to quantify the similarity in spatial distribution between two sets of point clouds, thereby providing a reliable reference standard for point cloud merging.

[0084] Specifically, the chamfer distance is the sum of the bidirectional nearest neighbor distances between two point cloud sets, comprehensively measuring the degree of geometric overlap. For the initial point cloud to be merged, it and the already merged instance point clouds are input into the calculation module. The nearest neighbor points between the two sets of point clouds are found point by point, and the squares of the nearest neighbor distances are accumulated as the chamfer distance. The calculation result of the first chamfer distance reflects the geometric similarity between the initial point cloud and the merged point cloud in 3D space. By setting a predetermined threshold, it can be ensured that only initial point clouds with chamfer distances less than the threshold are eligible for the next merging step.

[0085] Step 302: When the distance of the first chamfer is determined to be less than the preset threshold, the initial point cloud is merged with the currently merged instance point cloud.

[0086] In the multi-round iterative merging process, in order to ensure the geometric consistency between the initial point cloud and the currently merged instance point cloud, step 302 merges the initial point cloud and the currently merged instance point cloud when it is determined that the first chamfer distance is less than a preset threshold. The purpose of this step is to gradually integrate the point cloud fragments that meet the conditions into the current instance point cloud through accurate geometric similarity judgment, thereby ensuring the integrity and consistency of the point cloud.

[0087] Specifically, when the calculated first chamfer distance meets a preset threshold, it indicates that the geometry of the initial point cloud highly overlaps with or is similar to the merged instance point cloud in 3D space. At this point, through a geometric merging operation, all points from the initial point cloud are directly added to the dataset of the merged instance point cloud. During the merging process, the geometric features of the instance point cloud, such as its spatial extent and distribution density, are updated in real time to ensure that the merged instance point cloud accurately reflects the geometric characteristics of the current object.

[0088] Meanwhile, in order to avoid data redundancy or the introduction of duplicate information, the merging operation also needs to perform deduplication. For example, by checking the spatial location of each point, duplicate points that already exist in the instance point cloud are excluded, ensuring that the final merged instance point cloud is both complete and concise.

[0089] Step 303: Repeat the above merging process and update the point cloud of the currently merged instances after each round of merging.

[0090] Specifically, in the current round, based on the judgment and merging operation in step 302, the initial point clouds that meet the first chamfer distance threshold condition are merged one by one with the currently merged instance point clouds, and the geometric data of the merged instance point clouds are updated in real time. Whenever an initial point cloud is merged, the updated instance point cloud immediately participates in the merging process of subsequent initial point clouds, ensuring that each merge is based on the latest instance point cloud state for geometric judgment and integration.

[0091] After one round of merging, the coverage of the merged instance point cloud is further expanded, and all merged initial point clouds are removed from the pending set. In the next iteration, the first chamfer distance between the remaining initial point clouds and the merged instance point clouds is recalculated, and point clouds that meet the geometric consistency condition are selected to continue the merging operation. Through this iterative process, it is ensured that all initial point clouds that meet the conditions can be gradually integrated into the instance point clouds.

[0092] Step 304: When it is determined that there is no new initial point cloud that satisfies the condition that the first chamfer distance is less than the preset threshold, the iteration ends and the final instance point cloud of each object to be evaluated is obtained.

[0093] Specifically, after each round of merging, the system recalculates the first chamfer distance between all initial point clouds that have not yet participated in the merging and the currently merged instance point cloud. If, in this round of calculation, no initial point cloud satisfies the condition that the chamfer distance is less than the threshold, it means that all point cloud fragments that meet the geometric consistency condition have been integrated into the current instance point cloud. At this point, the iteration process terminates, and the final instance point cloud is output.

[0094] This termination condition ensures that the iterative merging process integrates all valid geometric information while avoiding unnecessary calculations and judgments on irrelevant or low-relevance point cloud fragments. In this way, the final generated instance point cloud not only fully represents the geometric characteristics of the object being evaluated but also has high computational efficiency.

[0095] Furthermore, the specific steps for obtaining the template point cloud corresponding to the category of each object to be evaluated are described in detail.

[0096] The method for obtaining template point clouds involves processing and optimizing multiple sample point clouds for each category to generate an average template point cloud that represents the geometric features of that category. First, a sufficient number of sample point clouds need to be collected. These point clouds can be obtained from 3D scanning equipment or publicly available point cloud datasets, ensuring coverage of the typical shape features of the category. These sample point clouds may initially have different coordinate systems, orientations, or point counts, therefore requiring a series of preprocessing operations, including alignment, normalization, and sampling, to ensure consistency and comparability of the point clouds.

[0097] In the preprocessing stage, the Iterative Closest Point (ICP) algorithm is used to align the orientation of the sample point cloud with the reference orientation, unify the coordinate system, and eliminate the influence of pose differences. Subsequently, the point cloud is uniformly sampled using a far-point sampling method to ensure that each point cloud has the same number of points and retains its core geometric features. At the same time, the geometric range of the point cloud is normalized and adjusted so that its size is adapted to a fixed standard range, thereby providing a consistent input basis for subsequent processing.

[0098] After preprocessing, a representative point cloud is selected from the sample point clouds as the initial template, or an initial template point cloud is generated by calculating the center of the point set of all sample point clouds. Next, an iterative optimization method is used to progressively improve the shape of the template point cloud so that it fully reflects the overall features of the category. In each iteration, a point correspondence is established based on the geometric distance between the template point cloud and each sample point cloud, pairing each point in the sample point cloud with the nearest point in the template point cloud. Then, the average position of all corresponding points is calculated, and the positions of corresponding points in the template point cloud are updated, gradually making the geometry of the template point cloud closer to the core features of the category. This iterative process continues until the change in the geometric position of the template point cloud is less than a set threshold or the maximum number of iterations is reached, thus generating the final converged template point cloud.

[0099] The generated template point cloud is an average shape representation of the category and can be used for subsequent shape recovery tasks. During storage, the template point cloud is associated with the category label to build a template point cloud library, so that the shape recovery network can call these template point clouds during prediction as geometric priors to complete incomplete instance point clouds. Through this method, the template point cloud can not only reflect the core geometric features of the category, but also be extended to shape variations within the category through Gaussian distribution or other deformation operations, thereby enhancing its generalization ability.

[0100] Step 102: Input the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by the instance point cloud samples and the corresponding real point cloud labels.

[0101] In dense and cluttered scenes, even if a complete instance point cloud is obtained through iterative merging of multi-view point clouds, accurate shape completion still requires further processing combined with category geometric priors. Therefore, step 102 inputs the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters output by the shape restoration network, providing crucial support for the final shape completion and predicted point cloud generation. This process utilizes the observation information of the instance point cloud and the category prior geometric information of the template point cloud, effectively fusing the features of both through a deep learning model, thus solving the problems of information loss and geometric incompleteness in shape completion.

[0102] In one possible implementation, the shape restoration network includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model. (Refer to...) Figure 5 , Figure 5 This is the fourth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention, as shown below. Figure 5 As shown, step 102 specifically includes steps 401-404:

[0103] Step 401: Input the instance point cloud of each object to be evaluated into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to extract global or basic features from the input point cloud data.

[0104] In dense and cluttered scenes, instance point clouds are obtained by merging multi-view data, primarily reflecting the geometry of objects under actual observation conditions. However, due to incompleteness and noise in the observation data, instance point clouds alone are insufficient for direct shape reconstruction. Therefore, step 401 involves inputting the instance point cloud into a basic point cloud feature extraction model to extract global or basic-level instance features, providing core geometric information from the observation data for subsequent shape reconstruction. The purpose of this step is to extract key features from the instance point cloud, ensuring that the model can understand and utilize the overall geometric distribution of the point cloud.

[0105] Specifically, the basic point cloud feature extraction model is a deep learning model (such as PointNet) that takes the 3D coordinate data of an instance point cloud as input. The model extracts basic features from the instance point cloud through point-by-point operations and a global pooling mechanism. For example, the basic point cloud feature extraction model applies a multilayer perceptron (MLP) to each point in the point cloud to extract the local features of each point, and then uses global max pooling to aggregate the features of all points into a global feature vector. This global feature vector reflects the overall geometric characteristics of the instance point cloud, including its spatial distribution and shape contour.

[0106] In this way, the basic point cloud feature extraction model can deeply encode the observed geometric information of instance point clouds, and the output instance features provide a high-quality basic representation for subsequent feature fusion and shape restoration. This feature extraction method can not only effectively handle the irregularity of point clouds, but also capture the main geometric shape of objects at a global scale.

[0107] Step 402: Input the template point cloud of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on the hierarchical or regional division of point cloud data.

[0108] In the shape restoration process of dense and cluttered scenes, template point clouds, as geometric priors for each category, are crucial for achieving incompleteness and accurate shape estimation of instance point clouds. However, template point clouds often represent the average shape of a category, containing category-level geometric features and detailed distributions. Therefore, step 402 inputs the template point cloud into a hierarchical point cloud feature extraction model to extract multi-level template features, providing rich prior information for subsequent feature fusion with instance point clouds. The purpose of this step is to perform hierarchical feature representation of the template point cloud, ensuring that the model can fully understand the geometric details and overall structure of the template point cloud.

[0109] Specifically, hierarchical point cloud feature extraction models (such as PointNet++) employ a hierarchical processing approach to extract geometric information from template point clouds at multiple levels. First, the template point cloud is divided into multiple local regions, and local features are extracted from each region through sampling and neighborhood search operations. Subsequently, these local features are aggregated and abstracted layer by layer to generate a high-level feature representation. The advantage of hierarchical extraction is that it can simultaneously capture both the local geometric details and the global geometric distribution of the template point cloud, reflecting both the detailed texture of the template point cloud and preserving the overall characteristics of prior class information.

[0110] Through hierarchical feature extraction, the complex geometric information of the template point cloud is effectively encoded into a multi-level feature vector. These features can provide category-specific geometric references for the instance point cloud. For example, template features can help the model complete the shape of the object through geometric priors when instance point cloud information is missing, and improve the model's generalization ability to unseen object categories.

[0111] Step 403: Extract the fusion features of each object to be evaluated from the instance point cloud and template point cloud of each object to be evaluated through the cross-attention mechanism.

[0112] In the shape restoration process, instance point clouds and template point clouds provide observational geometric information and category prior information, respectively. The effective combination of these two is crucial for achieving high-precision shape estimation. However, instance point clouds may suffer from occlusion or incompleteness, while template point clouds, as prior information, need to be dynamically adjusted based on the actual observational characteristics of the instance point clouds. Therefore, step 403 uses a cross-attention mechanism to fuse the features of the instance point clouds and template point clouds, generating fused features for each object to be evaluated. The purpose of this step is to fully utilize the category prior information of the template point clouds to compensate for the observational deficiencies of the instance point clouds, while simultaneously generating feature representations with both semantic and geometric constraints through the fusion operation.

[0113] Specifically, the cross-attention mechanism establishes a dynamic relationship between instance features and template features. First, the feature vector of the instance point cloud (generated by the base point cloud feature extraction model) is used as the query, and the feature vector of the template point cloud (generated by the hierarchical point cloud feature extraction model) is used as the key and value. Through attention calculation, the model dynamically extracts relevant information from the template point cloud features based on the geometric requirements of the instance point cloud features. The attention weights are assigned based on the similarity between the two sets of features, ensuring that highly relevant geometric and category information is enhanced.

[0114] Through the cross-attention mechanism, the fused features not only retain the observation information of the instance point cloud but also incorporate the class prior characteristics of the template point cloud, enabling the fused features to effectively reflect the actual observed shape and potential geometric completion of the object. After generation, the fused features serve as an intermediate representation for the shape restoration network, providing more semantically and geometrically meaningful input for subsequent shape restoration parameter prediction.

[0115] Step 404: Input the template features, fusion features and instance features of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network.

[0116] In the shape restoration network, fused features, template features, and instance features reflect the observation information of the instance point cloud, the prior category information of the template point cloud, and the dynamic combination characteristics of the two, respectively. To ultimately predict the shape restoration parameters of each object to be evaluated, step 404 generates shape restoration parameters, including the scaling factor and point-by-point offset of the template point cloud, by inputting the template features, fused features, and instance features into the hierarchical point cloud feature extraction model. The core purpose of this step is to combine multiple feature information to enable the model to accurately estimate the shape adjustment parameters of the template point cloud, thereby filling in the observational gaps in the instance point cloud and generating a high-precision predicted point cloud.

[0117] Specifically, template features, fused features, and instance features are merged into a joint feature vector. This feature vector contains both the geometric observation information of the instance point cloud and the category-level geometric prior of the template point cloud, as well as the dynamic fusion relationship between the two. After the joint feature vector is input into the hierarchical point cloud feature extraction model, the model performs further deep encoding on the feature vector through layer-by-layer feature extraction and convergence, capturing the multi-scale geometric information and potential shape characteristics of the object.

[0118] During feature extraction, the model predicts shape recovery parameters, including the scaling factor and point-by-point offset of the template point cloud, through a series of fully connected layers and nonlinear activation functions. The scaling factor reflects the global geometric adjustment ratio of the template point cloud relative to the instance point cloud, while the point-by-point offset is used to refine and correct the local geometry of the template point cloud. By predicting these two sets of parameters, the model can dynamically adjust the shape of the template point cloud to make it highly consistent with the observed data of the instance point cloud.

[0119] It should be noted that the specific training method of the shape recovery network will be described in detail in the following embodiments.

[0120] Step 103: Transform the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated, and obtain the predicted point cloud of each object to be evaluated.

[0121] During shape restoration, shape restoration parameters provide geometric adjustment information for the template point cloud, including scaling factors and point-by-point offsets. However, to transform these parameters into the specific 3D shape of an object, step 103 generates a predicted point cloud for each object to be evaluated by transforming the template point cloud. The core objective of this step is to utilize the class prior of the template point cloud and dynamically adjust its shape using shape restoration parameters to ensure high consistency with the observation information of the instance point cloud, thereby achieving shape completion and high-precision prediction.

[0122] In one possible implementation, refer to Figure 6 , Figure 6 This is the fifth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention, as shown below. Figure 6 As shown, step 103 specifically includes steps 501-502:

[0123] Step 501: Multiply the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated.

[0124] Step 502: Add the scaled point cloud of each object to be evaluated to the point-by-point offset of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0125] In step 501, the template point cloud is first globally adjusted according to the scaling factor in the shape restoration parameters. The scaling factor contains a scale parameter for the three coordinate axes (x, y, z) of the template point cloud, representing the scale transformation requirements of the template point cloud in different directions. The coordinates of each point in the template point cloud are multiplied by the scaling factor of the corresponding axis to generate a scaled point cloud. Through this operation, the global geometric characteristics of the template point cloud are adjusted, making its overall size and scale closer to the observation characteristics of the instance point cloud, thus laying the foundation for subsequent local correction.

[0126] In step 502, the scaled point cloud is further refined and corrected using point-by-point offsets. The point-by-point offsets, another part of the shape restoration parameters, record the local adjustment requirements for each point in the template point cloud. The coordinates of each point in the scaled point cloud are added to the corresponding point-by-point offset to generate the final predicted point cloud. This point-by-point correction operation compensates for differences in local detail in the template point cloud, ensuring that the generated predicted point cloud retains the geometric priors of the template point cloud while closely matching the observation characteristics of the instance point cloud.

[0127] In summary, the specific calculation for predicting point clouds can be expressed by the following formula:

[0128] , ;

[0129] In the formula, Let be the predicted point cloud of the k-th object to be evaluated. For the template point cloud corresponding to the k-th object to be evaluated, Let be the scaling factor for the k-th object to be evaluated. Let be the point-by-point offset of the k-th object to be evaluated. Let x be the offset of the k-th object to be evaluated on the x-axis. Let be the offset of the k-th object to be evaluated on the y-axis. This represents the offset of the k-th object to be evaluated on the z-axis.

[0130] In one possible implementation, refer to Figure 7 , Figure 7 This is the sixth flowchart of the shape estimation method for dense and cluttered objects provided by the present invention, as shown below. Figure 7 As shown, the training method for the shape recovery network includes steps 601-606:

[0131] Step 601: Obtain instance point cloud samples, template point cloud samples, and real point cloud labels.

[0132] In the training process of the shape restoration network, obtaining high-quality training data is fundamental to the effective learning of the model. Step 601 provides the necessary data support for the training process by acquiring instance point cloud samples, template point cloud samples, and real point cloud labels. The purpose of this step is to construct a complete and representative training dataset, enabling the shape restoration network to learn the geometric relationship between instance point clouds and template point clouds, as well as the matching relationship between predicted point clouds and real point clouds.

[0133] Specifically, instance point cloud samples are point cloud fragments generated from multi-view data. These fragments are iteratively merged to form a complete object point cloud, reflecting the geometric characteristics of the object under actual observation conditions. Template point cloud samples are geometric prior representations for each category, generated by averaging and optimizing multiple category sample point clouds, reflecting the global geometric features and detailed distribution of the category. Real point cloud labels are obtained through high-precision 3D scanning equipment or manually labeled and verified to ensure their accuracy as reference ground truth. These real point cloud labels provide supervisory signals to the network, enabling it to effectively correct prediction results.

[0134] During data acquisition, it is necessary to ensure the diversity and balance of instance point cloud samples and template point cloud samples to cover different shapes and categories of the target object. Simultaneously, the real point cloud labels need to correspond one-to-one with the instance and template point clouds and be precisely aligned to ensure that the supervision signals during training can accurately guide network optimization.

[0135] Step 602: Input the instance point cloud sample and the template point cloud sample into the initial network to obtain the point-by-point offset sample and scaling factor sample output by the initial network.

[0136] Specifically, instance point cloud samples and template point cloud samples are first input into the front-end module of the shape restoration network, which includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model. The basic point cloud feature extraction model extracts observed geometric features from the instance point cloud samples, while the hierarchical point cloud feature extraction model extracts class prior geometric features from the template point cloud samples. These features are then fused through a cross-attention mechanism to generate fused features, representing the combined geometric relationship between the instance point cloud and the template point cloud.

[0137] After feature fusion, the initial network uses these features for further processing to predict the shape restoration parameters of the template point cloud samples, namely, point-by-point offset samples and scaling factor samples. Point-by-point offset samples represent the adjustment requirements of the template point cloud in local geometry, while scaling factor samples represent the adjustment requirements of the template point cloud in global size. These samples not only express the geometric relationship between the template point cloud and the instance point cloud but also provide quantifiable adjustment criteria for the subsequent generation of predicted point clouds.

[0138] Step 603: Transform the template point cloud sample based on the point-by-point offset sample and the scaling factor sample to obtain the predicted point cloud sample.

[0139] Specifically, the template point cloud sample is first globally adjusted based on the scaling factor sample. The scaling factor sample provides the global scaling ratio of the template point cloud on the three coordinate axes. The coordinates of each point in the template point cloud sample are multiplied by the scaling factor of the corresponding axis to generate the scaled point cloud sample. This step achieves global geometric adjustment of the template point cloud, making its overall size closer to the observation characteristics of the instance point cloud sample.

[0140] Based on the scaled point cloud samples, point-by-point offset samples are used to further refine and adjust the local geometry of the point cloud. The final predicted point cloud sample is generated by adding the coordinates of each point in the scaled point cloud sample to its corresponding point-by-point offset. This point-by-point offset operation can finely correct for differences in the local geometry of the template point cloud, ensuring that the predicted point cloud sample not only retains the prior class features of the template point cloud but also closely matches the observation details of the instance point cloud sample.

[0141] Step 604: Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label.

[0142] Specifically, the second chamfer distance is a metric that measures the geometric similarity between a predicted point cloud sample and its ground truth point cloud label. Its calculation involves distance measurements in two directions: First, for each point in the predicted point cloud sample, the nearest neighbor in the ground truth point cloud label is found, and the squared Euclidean distance between the two points is calculated. The average of all these squared distances is taken as the error in the first direction. Second, for each point in the ground truth point cloud label, the nearest neighbor in the predicted point cloud sample is found again, the squared Euclidean distance between the two points is calculated, and the average is taken as the error in the second direction. Finally, the errors in the two directions are added together to obtain the value of the second chamfer distance.

[0143] Through this process, the second chamfer distance can comprehensively reflect the geometric differences between the predicted point cloud sample and the real point cloud label in 3D space. Due to its bidirectional matching characteristics, this distance can simultaneously assess whether the predicted point cloud fully covers the real point cloud and whether there is redundancy or unnecessary offset, thus providing a comprehensive geometric similarity assessment.

[0144] Step 605: Construct the loss function based on the second chamfer distance and the point-by-point offset samples.

[0145] During the training of the shape restoration network, relying solely on the geometric similarity between the predicted point cloud samples and the real point cloud labels is insufficient to ensure the global accuracy and local detail precision of the model's generated results. To further constrain the network output, step 605 constructs a comprehensive loss function by combining the second chamfer distance and point-by-point offset samples, enabling the network to achieve a balance between global consistency and local optimization in shape restoration. The purpose of this step is to provide multi-dimensional supervision signals for network optimization, thereby improving the prediction quality of shape restoration parameters.

[0146] Specifically, the loss function consists of two parts: one is the second chamfer distance, which quantifies the geometric difference between the predicted point cloud samples and the ground truth point cloud labels; the other is a regularization term for the point-by-point offset samples, which limits the magnitude of the offset and prevents the model from over-adjusting the template point cloud, causing the shape to deviate from the class prior. The second chamfer distance reflects the geometric similarity between the predicted point cloud and the ground truth point cloud through bidirectional nearest neighbor matching, while the regularization term for the point-by-point offset is calculated using the L2 norm, summing the squares of the offsets of all points and taking the mean, as a constraint on the local offset.

[0147] The loss function is expressed by the following formula:

[0148] ;

[0149] In the formula, For loss function, For real point cloud labels, To predict point cloud samples, The distance of the second chamfer. For point-by-point offset samples, It is the L2 norm. The sum of the L2 norms of the point-by-point offset samples is given, where n is the number of points in the predicted point cloud sample. This is a hyperparameter.

[0150] In this way, the loss function can optimize the overall geometric consistency of the predicted point cloud while constraining the magnitude of the point-by-point offset, thereby avoiding overfitting or unreasonable shape adjustments. This comprehensive constraint ensures that the model can match the global characteristics of the real point cloud while maintaining the local rationality of the generated point cloud during shape restoration.

[0151] Step 606: Train the initial network with the goal of minimizing the loss function to obtain the shape recovery network.

[0152] Specifically, the training process employs a supervised learning method, using instance point cloud samples and template point cloud samples as input, and real point cloud labels as supervision signals, based on a constructed loss function. The network parameters are optimized using backpropagation and gradient descent. In each training iteration, instance point cloud samples and template point cloud samples undergo multi-layer feature extraction and fusion by the initial network to generate point-by-point offset samples and scaling factor samples. These parameters are then used to transform the template point cloud samples to generate predicted point cloud samples.

[0153] Subsequently, the second chamfer distance is calculated between the predicted point cloud samples and the ground truth point cloud labels, while the constraint contribution of each point offset sample is calculated through a regularization term. These two parts together constitute the current value of the loss function. Using the backpropagation algorithm, the gradient of the loss function with respect to the network parameters is calculated, and optimization algorithms (such as Adam or SGD) are used to update the network weights, causing the loss function value to gradually decrease.

[0154] During training, the network continuously adjusts its weights to better generate predicted point clouds that match the real point cloud labels, while avoiding unreasonable shape adjustments caused by excessive point-by-point offsets. Through multiple rounds of iterative training, the network's predictive ability gradually improves, eventually enabling it to generate accurate shape recovery parameters from instance point cloud samples and template point cloud samples.

[0155] Through step 606, the shape restoration network optimizes its weights by minimizing the loss function in multiple iterations, successfully training a model with high-precision shape completion capabilities. The optimized network can simultaneously consider global geometric consistency and local detail correction, thereby achieving accurate shape estimation of dense and cluttered objects in complex scenes.

[0156] Reference Figure 8 , Figure 8 This is a schematic diagram of the shape estimation system for dense and cluttered objects provided by the present invention. The system includes:

[0157] The acquisition module is used to acquire instance point clouds of each object to be evaluated and to acquire template point clouds corresponding to the category of each object to be evaluated. Instance point clouds are used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and template point clouds are used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated.

[0158] The processing module is used to input the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels;

[0159] The transformation module is used to transform the template point cloud corresponding to each object to be evaluated based on the shape recovery parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated.

[0160] In one possible implementation, the acquisition module is further configured to acquire color images of a pile of objects consisting of multiple objects to be evaluated from multiple viewpoints, and acquire a depth image corresponding to each color image;

[0161] The processing module is also used to input each color image into the vocabulary segmentation model to obtain the instance segmentation mask and predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images from multiple perspectives;

[0162] The processing module is also used to combine the instance segmentation mask of each object to be evaluated in each view with the corresponding depth image to obtain the initial point cloud of each object to be evaluated in each view.

[0163] The processing module is also used to perform multiple rounds of iterative merging of initial point clouds with the same prediction category from different viewpoints to obtain instance point clouds of each object to be evaluated.

[0164] In one possible implementation, the processing module is further configured to:

[0165] In each iteration, the first chamfer distance between the initial point with the same predicted category under each different viewpoint and the currently merged instance point cloud is calculated;

[0166] When the distance to the first chamfer is determined to be less than the preset threshold, the initial point cloud is merged with the currently merged instance point cloud;

[0167] Repeat the above merging process and update the point cloud of currently merged instances after each round of merging;

[0168] When it is determined that no new initial point cloud satisfies the condition that the first chamfer distance is less than a preset threshold, the iteration ends, and the final instance point cloud of each object to be evaluated is obtained.

[0169] In one possible implementation, the processing module is further configured to:

[0170] The instance point cloud of each object to be evaluated is input into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to extract features at the global or basic level from the input point cloud data.

[0171] The template point cloud of each object to be evaluated is input into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on the hierarchical or regional division of point cloud data.

[0172] By using a cross-attention mechanism, the fusion features of each object to be evaluated are extracted from the instance point cloud and template point cloud of each object to be evaluated.

[0173] The template features, fusion features, and instance features of each object to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network.

[0174] In one possible implementation, the transformation module is further configured to:

[0175] Multiply the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated.

[0176] The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to obtain the predicted point cloud of each object to be evaluated.

[0177] In one possible implementation, the system further includes a model training module for:

[0178] Obtain instance point cloud samples, template point cloud samples, and real point cloud labels;

[0179] Input instance point cloud samples and template point cloud samples into the initial network to obtain point-by-point offset samples and scaling factor samples output by the initial network;

[0180] The template point cloud sample is transformed based on the point-by-point offset sample and the scaling factor sample to obtain the predicted point cloud sample.

[0181] Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label;

[0182] Based on the second chamfer distance and point-by-point offset samples, a loss function is constructed;

[0183] The initial network is trained with the goal of minimizing the loss function, resulting in a shape recovery network.

[0184] It should be noted that the shape estimation system for dense and cluttered objects provided by the present invention can execute the shape estimation method for dense and cluttered objects of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0185] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a shape estimation method for dense and cluttered objects. This method includes: acquiring an instance point cloud for each object to be evaluated, and acquiring a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained using instance point cloud samples and corresponding real point cloud labels; transforming the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0186] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by the computer, the computer is able to execute the shape estimation method for dense and cluttered objects provided in the above embodiments. The method includes: acquiring an instance point cloud of each object to be evaluated, and acquiring a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels; transforming the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0188] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by processor 910, is implemented to perform the shape estimation method for dense and cluttered objects provided in the above embodiments. The method includes: acquiring an instance point cloud of each object to be evaluated, and acquiring a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape restoration network to obtain shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained using instance point cloud samples and corresponding real point cloud labels; transforming the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

[0189] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A shape estimation method for densely packed, cluttered objects, characterized in that, include: Obtain the instance point cloud of each object to be evaluated, and obtain the template point cloud corresponding to the category of each object to be evaluated; The instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated. The instance point cloud and template point cloud of each object to be evaluated are input into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels. Transform the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated; the shape restoration parameters include a scaling factor and a point-by-point offset; the transformation of the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated specifically includes: Multiply the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated. The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

2. The shape estimation method for densely packed, cluttered objects according to claim 1, characterized in that, The acquisition of instance point clouds for each object to be evaluated specifically includes: Obtain color images of a pile of objects consisting of multiple objects to be evaluated from multiple viewpoints, and obtain a depth image corresponding to each color image; Each of the color images is input into the vocabulary segmentation model to obtain the instance segmentation mask and predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images from multiple perspectives; The instance segmentation mask of each object to be evaluated under each viewpoint is combined with the corresponding depth image to obtain the initial point cloud of each object to be evaluated under each viewpoint. Multiple rounds of iterative merging are performed on initial point clouds with the same prediction category from different viewpoints to obtain instance point clouds of each object to be evaluated.

3. The shape estimation method for densely packed, cluttered objects according to claim 2, characterized in that, The process of iteratively merging initial point clouds with the same prediction category from different viewpoints to obtain an instance point cloud for each object to be evaluated specifically includes: In each iteration, the first chamfer distance between the initial point with the same predicted category under each different viewpoint and the currently merged instance point cloud is calculated; When it is determined that the first chamfer distance is less than a preset threshold, the initial point cloud is merged with the currently merged instance point cloud; Repeat the above merging process and update the currently merged instance point cloud after each round of merging; When it is determined that no new initial point cloud satisfies the condition that the first chamfer distance is less than a preset threshold, the iteration ends, and the final instance point cloud of each object to be evaluated is obtained.

4. The shape estimation method for densely packed, cluttered objects according to claim 1, characterized in that, The shape restoration network includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model; The step of inputting the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network specifically includes: The instance point cloud of each object to be evaluated is input into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic-level feature extraction on the input point cloud data. The template point cloud of each object to be evaluated is input into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on the hierarchical or regional division of point cloud data. By using a cross-attention mechanism, the fusion features of each object to be evaluated are extracted from the instance point cloud and template point cloud of each object to be evaluated. The template features, fusion features, and instance features of each object to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network.

5. The shape estimation method for densely packed, cluttered objects according to claim 1, characterized in that, It also includes the training method for the shape recovery network: Obtain instance point cloud samples, template point cloud samples, and real point cloud labels; The instance point cloud sample and the template point cloud sample are input into the initial network to obtain the point-by-point offset sample and scaling factor sample output by the initial network. The template point cloud sample is transformed based on the point-by-point offset sample and the scaling factor sample to obtain the predicted point cloud sample. Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label; Based on the second chamfer distance and the point-by-point offset samples, a loss function is constructed; The initial network is trained with the goal of minimizing the loss function to obtain the shape recovery network.

6. The shape estimation method for densely packed, cluttered objects according to claim 5, characterized in that, The step of constructing a loss function based on the second chamfer distance and the point-by-point offset samples specifically includes: The loss function is expressed by the following formula: In the formula, L est Let S be the loss function, and S be the true point cloud label. For the predicted point cloud sample, ΔS is the second chamfer distance, ΔS is the point-by-point offset sample, ||| is the L2 norm, ||ΔS|| is the sum of the L2 norms of the point-by-point offset samples, n is the number of points in the predicted point cloud sample, and η is the hyperparameter.

7. A shape estimation system for densely packed, cluttered objects, characterized in that, include: The acquisition module is used to acquire the instance point cloud of each object to be evaluated, and to acquire the template point cloud corresponding to the category of each object to be evaluated; The instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated. The processing module is used to input the instance point cloud and template point cloud of each object to be evaluated into the shape restoration network to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network; the shape restoration network is trained by instance point cloud samples and corresponding real point cloud labels. The transformation module is used to transform the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated; the shape restoration parameters include a scaling factor and a point-by-point offset; the transformation of the template point cloud corresponding to each object to be evaluated according to the shape restoration parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated specifically includes: multiplying the scaling factor of each object to be evaluated with the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated; The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the shape estimation method for dense, cluttered objects as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the shape estimation method for dense and cluttered objects as described in any one of claims 1-6.

Images