Shape estimation method and system for dense and disordered objects

Abstract

The invention provides a shape estimation method and system for dense and messy objects, and belongs to the technical field of computer vision, and the method comprises the steps: obtaining an instance point cloud of each to-be-evaluated object, and obtaining a template point cloud corresponding to the category of each to-be-evaluated object; inputting the instance point cloud and the template point cloud of each to-be-evaluated object into a shape recovery network to obtain a shape recovery parameter of each to-be-evaluated object output by the shape recovery network; the shape recovery network is obtained by training instance point cloud samples and corresponding real point cloud labels; and transforming the template point cloud corresponding to each to-be-evaluated object according to the shape recovery parameter of each to-be-evaluated object to obtain a predicted point cloud of each to-be-evaluated object. According to the method, the observation information of the instance point cloud and the category geometric priori of the template point cloud are combined, and the shape recovery parameters are dynamically generated by using the shape recovery network, so that high-precision shape estimation of the to-be-evaluated object in the dense and disordered scene is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a shape estimation method and system for dense and cluttered objects. Background Art

[0002] With the development of automation and intelligent technology, robots are increasingly used in dense and cluttered environments, such as logistics sorting, industrial assembly, and warehouse management. In these scenarios, robots need to accurately identify and operate objects that are mutually occluded, overlapping, or partially visible in complex environments. The key to achieving this goal is to accurately estimate the three-dimensional shape of the target object.

[0003] However, traditional technologies usually rely on single-view image data or directly use point cloud data acquired by three-dimensional sensors for processing. In single-view methods, due to limited viewing angles and occlusion problems, the acquired point cloud data is often incomplete, which severely limits the accuracy of shape estimation. Although multi-view point cloud fusion methods can partially alleviate the problem of missing information, when dealing with multi-object scenes, the performance of existing methods is still unsatisfactory due to dense occlusion between objects and difficulty in separating point clouds. In addition, some technologies attempt to directly match target objects through predefined template shapes, but this method lacks the ability to dynamically model instantiated features and geometric details, resulting in poor generalization and difficulty in adapting to the diverse needs of unseen objects or categories.

[0004] Therefore, how to achieve complete and high-precision shape estimation of the object to be evaluated in dense and cluttered scenes has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a shape estimation method, system, electronic device and storage medium for dense and cluttered objects, so as to solve the defects in the prior art and realize complete and high-precision shape estimation of partially observed objects to be evaluated in dense and cluttered scenes.

[0006] The present invention provides a shape estimation method for dense and cluttered objects, comprising the following steps: Acquire an instance point cloud of each object to be evaluated, and acquire a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; Inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape recovery network, and obtaining the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by instance point cloud samples and corresponding real point cloud labels; The template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameter of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

[0007] According to a shape estimation method for dense and cluttered objects provided by the present invention, obtaining an instance point cloud of each object to be evaluated specifically includes: Acquire color images of an object stack composed of a plurality of the objects to be evaluated at multiple viewing angles, and acquire a depth image corresponding to each of the color images; Inputting each of the color images into a vocabulary segmentation model to obtain an instance segmentation mask and a predicted category of each of the objects to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images under multiple viewing angles; Combining the instance segmentation mask of each object to be evaluated at each viewing angle with the corresponding depth image to obtain an initial point cloud of each object to be evaluated at each viewing angle; The initial point clouds with the same prediction category under different viewing angles are iteratively merged for multiple rounds to obtain instance point clouds of each object to be evaluated.

[0008] According to a shape estimation method for dense and cluttered objects provided by the present invention, the initial point clouds with the same prediction category under different viewing angles are iteratively merged for multiple rounds to obtain instance point clouds of each object to be evaluated, specifically including: In each round of iteration, the first chamfer distance between the initial point with the same predicted category and the currently merged instance point cloud under each different viewing angle is calculated; When it is determined that the first chamfer distance is less than a preset threshold, merging the initial point cloud with the currently merged instance point cloud; Repeat the above merging process and update the currently merged instance point cloud after each round of merging is completed; When it is determined that there is no new initial point cloud that satisfies the condition that the first chamfer distance is less than the preset threshold, the iteration is terminated to obtain the final instance point cloud of each object to be evaluated.

[0009] According to a shape estimation method for dense and cluttered objects provided by the present invention, the shape recovery network includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model; the example point cloud and the template point cloud of each object to be evaluated are input into the shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network, specifically including: Inputting each instance point cloud of the object to be evaluated into the basic point cloud feature extraction model to obtain instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic level feature extraction on the input point cloud data; Inputting the template point cloud of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on stratification or regionalization of point cloud data; Extracting fusion features of each object to be evaluated from the instance point cloud and the template point cloud of each object to be evaluated through a cross attention mechanism; The template features, the fusion features and the instance features of each of the objects to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each of the objects to be evaluated output by the shape restoration network.

[0010] According to a shape estimation method for dense and cluttered objects provided by the present invention, the shape recovery parameters include a scaling factor and a point-by-point offset; the template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameters of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated, specifically including: Multiplying the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain a scaled point cloud of each object to be evaluated; The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

[0011] A shape estimation method for dense and cluttered objects provided by the present invention also includes a training method for the shape recovery network: Obtain instance point cloud samples, template point cloud samples and real point cloud labels; Input the instance point cloud samples and the template point cloud samples into an initial network to obtain point-by-point offset samples and scaling factor samples output by the initial network; Transforming the template point cloud samples according to the point-by-point offset samples and the scaling factor samples to obtain predicted point cloud samples; Calculating a second chamfer distance between the predicted point cloud sample and the real point cloud label; Constructing a loss function according to the second chamfer distance and the point-by-point offset samples; The initial network is trained with minimizing the value of the loss function as a training goal to obtain the shape recovery network.

[0012] According to a shape estimation method for dense and cluttered objects provided by the present invention, constructing a loss function according to the second chamfer distance and the point-by-point offset samples specifically includes: The loss function is expressed by the following formula: ; In the formula, is the loss function, is the real point cloud label, is the predicted point cloud sample, is the second chamfer distance, is the point-by-point offset sample, is the L2 norm, is the sum of the L2 norms of the point-by-point offset samples, n is the number of points in the predicted point cloud samples, is a hyperparameter.

[0013] The present invention also provides a shape estimation system for dense and cluttered objects, comprising the following modules: An acquisition module, used to acquire an instance point cloud of each object to be evaluated, and to acquire a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; A processing module, used for inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape recovery network, and obtaining the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is obtained by training the instance point cloud samples and the corresponding real point cloud labels; The transformation module is used to transform the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a shape estimation method for dense and cluttered objects as described in any one of the above methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the shape estimation method for dense and cluttered objects as described in any one of the above methods is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the shape estimation method for dense and cluttered objects as described in any one of the above is implemented.

[0017] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: By obtaining the instance point cloud and the template point cloud of the corresponding category of each object to be evaluated, a basis for combining observation and prior is provided for shape estimation. Among them, the instance point cloud reflects the three-dimensional point cloud data of the object under actual observation conditions and can capture its actual geometric distribution, but it may be incomplete due to occlusion and data loss in dense and cluttered environments. As the geometric prior of the category, the template point cloud provides a reliable reference benchmark for shape completion through optimization generation based on diversified samples, thereby making up for the problem of insufficient observation in the instance point cloud. The instance point cloud and template point cloud of each object to be evaluated are input into the shape recovery network, so that the network can use the observation information of the instance point cloud and the prior information of the template point cloud for feature fusion, and output the shape recovery parameters of each object. These parameters include the scaling factor and point-by-point offset of the template point cloud, which are used to globally adjust and locally correct the shape of the template point cloud, so that the template point cloud is consistent with the actual observed geometric characteristics of the instance point cloud. In this way, the shape recovery network can not only use the global geometric features of the template point cloud, but also dynamically adapt to the local observation characteristics of the instance point cloud, thereby achieving accurate estimation of the shape of objects in complex scenes. The shape recovery network is trained with instance point cloud samples and real point cloud labels to ensure that the network can learn the geometric mapping relationship between the instance point cloud and the template point cloud, and optimize its generation ability by minimizing the geometric error between the predicted point cloud and the real point cloud. By constructing a comprehensive loss function containing chamfer distance and point-by-point offset regularization terms, the training process constrains local adjustments while ensuring global geometric consistency, further improving the prediction accuracy of shape recovery parameters. The template point cloud is transformed according to the shape recovery parameters. First, the global shape of the template point cloud is adjusted by the scaling factor to make its size and proportion closer to the observed characteristics of the instance point cloud; then the template point cloud is locally corrected by the point-by-point offset to correct the detailed shape and finally generate a predicted point cloud. The predicted point cloud not only retains the category geometric characteristics of the template point cloud, but also fully adapts to the observed data of the instance point cloud, achieving complete and high-precision shape estimation of the object to be evaluated in dense and cluttered scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the shape estimation method for dense and cluttered objects provided by the present invention.

[0020] Figure 2 This is the second flowchart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0021] Figure 3 It is a schematic diagram of a shooting scene of a multi-view RGB-D camera provided by the present invention.

[0022] Figure 4 This is the third flow chart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0023] Figure 5 This is the fourth flow chart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0024] Figure 6 This is the fifth flow chart of the shape estimation method for dense and messy objects provided by the present invention.

[0025] Figure 7 This is the sixth flow chart of the shape estimation method for dense and cluttered objects provided by the present invention.

[0026] Figure 8 It is a structural schematic diagram of a shape estimation system for dense and messy objects provided by the present invention.

[0027] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that, in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the system or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0030] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] Combine the following Figure 1-Figure 9 The present invention describes a shape estimation method, system, electronic device and storage medium for dense and cluttered objects.

[0032] Figure 1 is one of the flow charts of the shape estimation method for dense and messy objects provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101: Obtain an instance point cloud of each object to be evaluated, and obtain a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to represent the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated.

[0033] In dense and cluttered scenes, due to object occlusion, overlap, and scene complexity, single-view point clouds often lack information and it is difficult to accurately extract complete point cloud data for a single object. Therefore, in step 101, by obtaining the instance point cloud of each object to be evaluated and combining it with the template point cloud corresponding to the category of each object to be evaluated, a complete and accurate geometric reference is provided for subsequent shape completion. In this process, the integrity and accuracy of the instance point cloud are ensured by obtaining data from multiple perspectives, using a vocabulary segmentation model for object segmentation and category prediction, and combining multiple rounds of iterative merging methods.

[0034] The following embodiments will explain in detail the method for obtaining the instance point cloud of each object to be evaluated and the template point cloud corresponding to the category of each object to be evaluated.

[0035] First, the specific steps for obtaining the instance point cloud of each object to be evaluated are described in detail.

[0036] In one possible implementation, refer to Figure 2 , Figure 2 FIG. 2 is a flow chart of the shape estimation method for dense and cluttered objects provided by the present invention. Figure 2 As shown, in step 101, the method for obtaining the instance point cloud of each object to be evaluated specifically includes steps 201-204: Step 201: obtaining color images of an object stack composed of a plurality of objects to be evaluated at a plurality of viewing angles, and obtaining a depth image corresponding to each color image.

[0037] Specifically, a multi-view RGB-D camera is set up to shoot the object pile, capturing the color image and the corresponding depth image at each view. For a specific shooting scene diagram, please refer to Figure 3 , Figure 3 It is a schematic diagram of a shooting scene of a multi-view RGB-D camera provided by the present invention.

[0038] The color image records the texture and color features of the scene, which are crucial for subsequent object segmentation and category prediction through the vocabulary segmentation model. At the same time, the depth image records the spatial depth information of the object surface, which together with the color image provides the two-dimensional and three-dimensional data basis required for point cloud generation. During the data acquisition process, by ensuring that the camera's position, angle, and viewing angle can cover the main area of ​​the object pile, occlusion and blind spots are minimized as much as possible, thereby improving the integrity and quality of the acquired point cloud data.

[0039] Step 202: input each color image into a vocabulary segmentation model to obtain an instance segmentation mask and a predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is obtained by fine-tuning sample color images under multiple viewing angles; Specifically, the word segmentation model is fine-tuned based on sample color images from multiple viewpoints. The model is able to recognize and segment unseen object categories through the semantic support of an open vocabulary.

[0040] In order to achieve accurate segmentation and category prediction of objects in dense and cluttered scenes, this solution uses the Detic open vocabulary segmentation model. Detic is a segmentation model pre-trained on the large-scale dataset ImageNet-21K, with rich external knowledge and strong generalization capabilities. With the support of open vocabulary, it can recognize multiple categories in complex scenes, and even unseen object categories can be effectively segmented and classified based on their semantic features.

[0041] In order to further improve the segmentation accuracy of the Detic model in specific scenarios, this solution fine-tunes it on a specific training set. The fine-tuning process is combined with an adapter layer. By adding a lightweight adapter module to the model, the vocabulary segmentation model obtained after fine-tuning the Detic model can better adapt to the unique visual features in dense and cluttered scenes, such as complex occlusions, fine-grained differences between similar categories, etc. As part of the model, the adapter layer can efficiently adjust the output of the vocabulary segmentation model so that its segmentation performance is more in line with the task requirements of this solution.

[0042] After each collected color image is input into the vocabulary segmentation model, the model performs semantic segmentation on the objects in each image and generates instance segmentation masks and corresponding category prediction labels. The instance segmentation mask is a two-dimensional pixel-level segmentation result for each object, which is used to represent the area range of each object in the image; the category prediction label provides the semantic category information of the object. The output of this combination of segmentation and classification can not only effectively handle complex occlusion and overlap problems in dense scenes, but also provide key semantic constraints for subsequent three-dimensional point cloud generation.

[0043] Step 203: combining the instance segmentation mask of each object to be evaluated at each viewing angle with the corresponding depth image to obtain an initial point cloud of each object to be evaluated at each viewing angle.

[0044] Specifically, at each viewing angle, the instance segmentation mask generated in step 202 is first pixel-aligned with the depth map corresponding to the color image. The area of ​​each segmentation mask corresponds to the position range of an object in the two-dimensional image, and the depth map provides depth information for the position in three-dimensional space. By using the intrinsic and extrinsic parameters of the camera, the pixel coordinates of each segmentation mask are combined with the depth value to be converted into three-dimensional space coordinates, and finally the initial point cloud of each object to be evaluated at the current viewing angle is generated. This process relies on the projection relationship formula of the camera, and the position of the object in three-dimensional space is calculated through pixel coordinates and depth values.

[0045] Through this method, the initial point cloud of each object in each perspective is accurately extracted, reflecting its local geometric characteristics in three-dimensional space. It is worth noting that since the segmentation mask is only mapped to the segmented area, the interference of background or other object information is avoided, thereby ensuring the purity and pertinence of the generated point cloud. In addition, the initial point cloud under multiple perspectives collects data from different angles, providing multi-perspective support for subsequent point cloud merging, which can effectively alleviate the problem of incomplete reconstruction caused by the lack of information from a single perspective.

[0046] Step 204: Perform multiple rounds of iterative merging on the initial point clouds with the same prediction category under different viewing angles to obtain instance point clouds of each object to be evaluated.

[0047] In dense and cluttered scenes, since the initial point cloud at each perspective can only reflect the local geometric information of the object, and the point cloud fragments at different perspectives often overlap or are inconsistent, in order to obtain a complete instance point cloud for each object to be evaluated, step 204 performs multiple rounds of iterative merging of the initial point clouds with the same prediction category at different perspectives to integrate the multi-perspective information into a unified point cloud representation, thereby solving the problem of missing shape information caused by local occlusion and single-perspective limitations.

[0048] In one possible implementation, refer to Figure 4 , Figure 4 FIG. 3 is a flow chart of the shape estimation method for dense and messy objects provided by the present invention. Figure 4 As shown, step 204 specifically includes steps 301-304: Step 301: In each iteration, the first chamfer distance between the initial point with the same predicted category and the currently merged instance point cloud under each different viewing angle is calculated.

[0049] In dense and cluttered scenes, the initial point cloud often comes from multiple perspectives, and there are positional offsets or geometric differences in point cloud fragments from different perspectives. In order to ensure that only point clouds with consistent geometric structures are integrated during the multi-round iterative merging process, step 301 calculates the first chamfer distance between the initial point cloud with the same predicted category and the currently merged instance point cloud under different perspectives as the basis for judging geometric consistency. The purpose of this step is to use the geometric characteristics of the chamfer distance to quantify the similarity of the two groups of point clouds in spatial distribution, thereby providing a reliable reference standard for point cloud merging.

[0050] Specifically, the chamfer distance is the sum of the two-way nearest neighbor distances between two point cloud sets, which can comprehensively measure the degree of geometric overlap of the point clouds. For the initial point cloud to be merged, it is input into the calculation module together with the merged instance point cloud, and the nearest neighbor points between the two sets of point clouds are found point by point, and the square of the nearest neighbor distance is accumulated as the chamfer distance. The calculation result of the first chamfer distance reflects the geometric similarity between the initial point cloud and the merged point cloud in three-dimensional space. By setting a predetermined threshold, it can be ensured that only the initial point cloud with a chamfer distance less than the threshold is eligible for the next step of merging.

[0051] Step 302: When it is determined that the first chamfer distance is less than a preset threshold, the initial point cloud is merged with the currently merged instance point cloud.

[0052] In the process of multiple rounds of iterative merging, in order to ensure the geometric consistency between the initial point cloud and the currently merged instance point cloud, when it is determined that the first chamfer distance is less than a preset threshold, the initial point cloud and the currently merged instance point cloud are merged in step 302. The purpose of this step is to gradually integrate the qualified point cloud fragments into the current instance point cloud through accurate geometric similarity judgment, thereby ensuring the integrity and consistency of the point cloud.

[0053] Specifically, when the calculation result of the first chamfer distance meets the preset threshold condition, it indicates that the geometric structure of the current initial point cloud is highly overlapped or similar to the merged instance point cloud in three-dimensional space. At this time, through the geometric merging operation, all points of the initial point cloud are directly added to the data set of the merged instance point cloud. During the merging process, the geometric features of the instance point cloud, such as the spatial range and distribution density of the point cloud, are updated in real time to ensure that the merged instance point cloud can accurately reflect the geometric characteristics of the current object.

[0054] At the same time, in order to avoid the introduction of data redundancy or duplicate information, the merging operation also requires deduplication processing, such as by checking the spatial position of each point and excluding duplicate points that already exist in the instance point cloud, to ensure that the final merged instance point cloud is both complete and concise.

[0055] Step 303: Repeat the above merging process and update the current merged instance point cloud after each round of merging is completed.

[0056] Specifically, in the current round, according to the judgment and merging operation in step 302, the initial point clouds that meet the first chamfer distance threshold condition are merged with the currently merged instance point clouds one by one, and the geometric data of the merged instance point clouds are updated in real time. Whenever an initial point cloud is merged, the updated instance point cloud immediately participates in the merging process of the subsequent initial point clouds to ensure that each merge is based on the latest instance point cloud state for geometric judgment and integration.

[0057] After completing a round of merging, the coverage of the current merged instance point cloud is further expanded, and all merged initial point clouds are removed from the set to be processed. In the next round of iteration, the first chamfer distance between the remaining initial point cloud and the merged instance point cloud is recalculated, and the point cloud that meets the geometric consistency conditions is screened out and the merging operation is continued. Through this repeated iteration method, it is ensured that all qualified initial point clouds can be gradually integrated into the instance point cloud.

[0058] Step 304: When it is determined that there is no new initial point cloud that satisfies the condition that the first chamfer distance is less than the preset threshold, the iteration is terminated to obtain the final instance point cloud of each object to be evaluated.

[0059] Specifically, after each round of merging is completed, the system recalculates the first chamfer distance for all initial point clouds that have not yet participated in the merging and the current merged instance point cloud. If in this round of calculation, no initial point cloud meets the condition that the chamfer distance is less than the threshold, it means that all point cloud fragments that meet the geometric consistency conditions have been integrated into the current instance point cloud. At this time, the iterative process is terminated and the final instance point cloud is output.

[0060] The setting of this termination condition ensures that the iterative merging process can integrate all valid geometric information, and avoids redundant calculations and judgments on irrelevant or low-relevance point cloud fragments. In this way, the final generated instance point cloud can not only fully characterize the geometric characteristics of the object to be evaluated, but also has high computational efficiency.

[0061] Furthermore, the specific steps of obtaining the template point cloud corresponding to the category of each object to be evaluated are described in detail.

[0062] The method for obtaining the template point cloud is to process and optimize multiple sample point clouds of each category to generate an average template point cloud that can represent the geometric features of the category. First, a sufficient number of sample point clouds need to be collected. These point clouds can come from 3D scanning equipment or public point cloud datasets to ensure that the typical shape features of the category are covered. These sample point clouds may have different coordinate systems, directions or numbers of points in the initial state, so they need to go through a series of preprocessing operations, including alignment, normalization and sampling, to ensure that the point clouds are consistent and comparable.

[0063] In the preprocessing stage, the iterative closest point algorithm (ICP) is used to align the direction of the sample point cloud with the reference direction, unify the coordinate system, and eliminate the influence caused by the posture difference. Subsequently, the point cloud is uniformly sampled by the far point sampling method to ensure that the number of points in each point cloud is the same and its core geometric features are retained. At the same time, the geometric range of the point cloud is normalized to adapt its size to a fixed standard range, thereby providing a consistent input basis for subsequent processing.

[0064] After preprocessing, a representative point cloud is selected from the sample point cloud as the initial template, or an initial template point cloud is generated by calculating the point set center of all sample point clouds. Next, the shape of the template point cloud is gradually improved through an iterative optimization method so that it can fully reflect the overall characteristics of the category. In each iteration, a point correspondence is established based on the geometric distance between the template point cloud and each sample point cloud, and each point in the sample point cloud is paired with the nearest point in the template point cloud. Then, the average position of all corresponding points is calculated, and the position of the corresponding points in the template point cloud is updated, gradually making the geometric shape of the template point cloud closer to the core features of the category. This iterative process continues until the geometric position change of the template point cloud is less than the set threshold or the maximum number of iterations is reached, thereby generating a final converged template point cloud.

[0065] The generated template point cloud is the average shape representation of the category and can be used for subsequent shape recovery tasks. During the storage process, the template point cloud is associated with the category label to build a template point cloud library so that the shape recovery network can call these template point clouds during the prediction process as geometric priors to complete the incomplete instance point cloud. Through this method, the template point cloud can not only reflect the core geometric features of the category, but also be extended to the shape changes within the category through Gaussian distribution or other deformation operations, thereby enhancing its generalization ability.

[0066] Step 102: Input the instance point cloud and template point cloud of each object to be evaluated into the shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by instance point cloud samples and corresponding real point cloud labels.

[0067] In dense and cluttered scenes, even if a complete instance point cloud is obtained through iterative merging of multi-view point clouds, accurate shape completion still needs to be further processed in combination with category geometry priors. Therefore, step 102 obtains the shape recovery parameters output by the shape recovery network by inputting the instance point cloud and template point cloud of each object to be evaluated into the shape recovery network, providing key support for the final shape completion and predicted point cloud generation. This process uses the observation information of the instance point cloud and the category prior geometry information of the template point cloud, and effectively integrates the features of the two through a deep learning model, solving the problems of missing information and incomplete geometry in shape completion.

[0068] In a possible implementation, the shape recovery network includes a basic point cloud feature extraction model and a hierarchical point cloud feature extraction model. Figure 5 , Figure 5 FIG. 4 is a flow chart of a method for estimating the shape of dense and cluttered objects provided by the present invention. Figure 5 As shown, step 102 specifically includes steps 401-404: Step 401: Input the instance point cloud of each object to be evaluated into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic level feature extraction on the input point cloud data.

[0069] In dense and cluttered scenes, instance point clouds are obtained by merging multi-view data, which mainly reflect the geometric shape of objects under actual observation conditions. However, due to the incompleteness and noise of the observation data, relying solely on instance point clouds is not enough to directly complete shape recovery. Therefore, step 401 extracts global or basic level instance features from the instance point cloud by inputting it into the basic point cloud feature extraction model, providing the core geometric information of the observation data for subsequent shape recovery. The purpose of this step is to extract the key features of the instance point cloud and ensure that the model can understand and utilize the overall geometric distribution of the point cloud.

[0070] Specifically, the basic point cloud feature extraction model is a deep learning model (such as PointNet), whose input is the three-dimensional coordinate data of the instance point cloud. The model extracts its basic features from the instance point cloud through point-by-point operations and global aggregation mechanisms. For example, the basic point cloud feature extraction model applies a multi-layer perceptron (MLP) to each point in the point cloud to extract the local features of each point, and then aggregates the features of all points into a global feature vector through a global maximum pooling operation. This global feature vector can reflect the overall geometric characteristics of the instance point cloud, including its spatial distribution, shape contours, etc.

[0071] In this way, the basic point cloud feature extraction model can deeply encode the observed geometric information of the instance point cloud, and the output instance features provide a high-quality basic representation for subsequent feature fusion and shape recovery. This feature extraction method can not only effectively handle the irregularity of the point cloud, but also capture the main geometric shape of the object at a global scale.

[0072] Step 402: Input the template point cloud of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on stratification or regionalization of point cloud data.

[0073] In the shape recovery process of dense and cluttered scenes, the template point cloud, as the geometric prior for each category, is the key to completing the instance point cloud and accurately estimating the shape. However, the template point cloud is often a representation of the average shape of the category, which contains the geometric features and detail distribution at the category level. Therefore, step 402 extracts multi-level template features from the template point cloud by inputting it into a hierarchical point cloud feature extraction model, thereby providing rich prior information for subsequent feature fusion with the instance point cloud. The purpose of this step is to perform a hierarchical feature representation of the template point cloud to ensure that the model can fully understand the geometric details and overall structure of the template point cloud.

[0074] Specifically, hierarchical point cloud feature extraction models (such as PointNet++) use a hierarchical processing method to perform multi-level extraction of geometric information of template point clouds. First, the template point cloud is divided into multiple local regions, and local features are extracted from each region through sampling and neighborhood search operations. Subsequently, these local features are aggregated and abstracted layer by layer to generate high-level feature representations. The characteristic of hierarchical extraction is that it can simultaneously capture the local geometric details and global geometric distribution of the template point cloud, reflecting the detailed texture of the template point cloud while retaining the overall characteristics of the category prior.

[0075] Through the hierarchical feature extraction method, the complex geometric information of the template point cloud is effectively encoded into a multi-level feature vector, which can provide category-specific geometric references for the instance point cloud. For example, the template feature can help the model complete the shape of the object through geometric priors when the instance point cloud information is missing, and improve the model's generalization ability for unseen object categories.

[0076] Step 403: extracting fusion features of each object to be evaluated from the instance point cloud and template point cloud of each object to be evaluated through a cross attention mechanism.

[0077] In the shape recovery process, the instance point cloud and the template point cloud provide observation geometry information and category prior information respectively. The effective combination of the two is the key to achieve high-precision shape estimation. However, the instance point cloud may be occluded or incomplete, and the template point cloud, as prior information, needs to be dynamically adjusted according to the actual observation characteristics of the instance point cloud. Therefore, step 403 fuses the features of the instance point cloud and the template point cloud through a cross-attention mechanism to generate fused features for each object to be evaluated. The purpose of this step is to make full use of the category prior information of the template point cloud to supplement the insufficient observation of the instance point cloud, and at the same time generate a feature representation with both semantic and geometric constraints through the fusion operation.

[0078] Specifically, the cross-attention mechanism establishes a dynamic association between instance features and template features. First, the feature vector of the instance point cloud (generated by the basic point cloud feature extraction model) is used as the query (Query), and the feature vector of the template point cloud (generated by the hierarchical point cloud feature extraction model) is used as the key (Key) and value (Value). Through attention calculation, the model dynamically extracts relevant information from the template point cloud features according to the geometric requirements of the instance point cloud features. The allocation of attention weights is based on the similarity between the two sets of features to ensure that highly relevant geometric and category information is strengthened.

[0079] Under the action of the cross-attention mechanism, the fused features not only retain the observation information of the instance point cloud, but also combine the category prior characteristics of the template point cloud, so that the fused features can effectively reflect the actual observed shape and potential geometric completion of the object. After the fused features are generated, as the intermediate representation of the shape recovery network, they can provide more semantically and geometrically meaningful input for the subsequent shape recovery parameter prediction.

[0080] Step 404: Input the template features, fusion features and instance features of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each object to be evaluated output by the shape restoration network.

[0081] In the shape recovery network, the fusion features, template features, and instance features respectively reflect the observation information of the instance point cloud, the category prior information of the template point cloud, and the dynamic combination characteristics of the two. In order to finally predict the shape recovery parameters of each object to be evaluated, step 404 generates shape recovery parameters, including the scaling factor and point-by-point offset of the template point cloud, by inputting the template features, fusion features, and instance features into the hierarchical point cloud feature extraction model. The core purpose of this step is to combine multiple feature information so that the model can accurately estimate the shape adjustment parameters of the template point cloud to complete the observation loss of the instance point cloud and generate a high-precision predicted point cloud.

[0082] Specifically, the template features, fusion features, and instance features are combined into a joint feature vector, which contains both the geometric observation information of the instance point cloud and the category-level geometric prior of the template point cloud as well as the dynamic fusion relationship between the two. After the joint feature vector is input into the hierarchical point cloud feature extraction model, the model further deeply encodes the feature vector through layer-by-layer feature extraction and aggregation to capture the multi-scale geometric information and potential shape characteristics of the object.

[0083] During the feature extraction process, the model predicts shape recovery parameters, including the scaling factor and point-by-point offset of the template point cloud, through a series of fully connected layers and nonlinear activation functions. The scaling factor reflects the global geometric adjustment ratio of the template point cloud relative to the instance point cloud, while the point-by-point offset is used to refine the local geometric shape of the template point cloud. Through the prediction of these two sets of parameters, the model can dynamically adjust the shape of the template point cloud to make it highly consistent with the observed data of the instance point cloud.

[0084] It should be noted that the specific training method of the shape recovery network will be described in detail in the subsequent embodiments.

[0085] Step 103: transforming the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

[0086] In the shape recovery process, the shape recovery parameters provide geometric adjustment information of the template point cloud, including scaling factors and point-by-point offsets. However, in order to convert these parameters into a specific three-dimensional shape of the object, step 103 transforms the template point cloud to generate a predicted point cloud for each object to be evaluated. The core purpose of this step is to use the category prior of the template point cloud and dynamically adjust its shape through the shape recovery parameters so that it is highly consistent with the observed information of the instance point cloud, thereby achieving shape completion and high-precision prediction.

[0087] In one possible implementation, refer to Figure 6 , Figure 6 FIG. 5 is a flowchart of a method for estimating the shape of dense and cluttered objects provided by the present invention. Figure 6 As shown, step 103 specifically includes steps 501-502: Step 501: multiplying the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain the scaled point cloud of each object to be evaluated.

[0088] Step 502: Add the scaled point cloud of each object to be evaluated and the point-by-point offset of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

[0089] In step 501, the template point cloud is first globally adjusted according to the scaling factor in the shape recovery parameters. The scaling factor contains the scale parameters for the three coordinate axes (x, y, z) of the template point cloud, which indicates the scale transformation requirements of the template point cloud in different directions. The coordinates of each point of the template point cloud are multiplied by the scaling factor of the corresponding axis to generate a scaled point cloud. Through this operation, the global geometric characteristics of the template point cloud are adjusted so that its overall size and proportion are closer to the observed characteristics of the instance point cloud, thereby laying the foundation for subsequent local correction.

[0090] In step 502, the scaled point cloud is further refined and corrected by point-by-point offsets. Point-by-point offsets are another part of the shape recovery parameters, which record the local adjustment requirements of each point in the template point cloud. The coordinates of each point in the scaled point cloud are added to the corresponding point-by-point offsets to generate the final predicted point cloud. This point-by-point correction operation can compensate for the differences in local details of the template point cloud, so that the generated predicted point cloud not only retains the geometric priors of the template point cloud, but also can highly fit the observation characteristics of the instance point cloud.

[0091] In summary, the specific calculation of the predicted point cloud can be expressed by the following formula: , ; In the formula, is the predicted point cloud of the kth object to be evaluated, is the template point cloud corresponding to the kth object to be evaluated, is the scaling factor of the kth object to be evaluated, is the point-by-point offset of the kth object to be evaluated, is the offset of the kth object to be evaluated on the x-axis, is the offset of the kth object to be evaluated on the y-axis, is the offset of the kth object to be evaluated on the z-axis.

[0092] In one possible implementation, refer to Figure 7 , Figure 7 FIG. 6 is a flow chart of a method for estimating the shape of dense and cluttered objects provided by the present invention. Figure 7 As shown, the training method of the shape recovery network includes steps 601-606: Step 601: Obtain instance point cloud samples, template point cloud samples and real point cloud labels.

[0093] In the training process of the shape recovery network, obtaining high-quality training data is the basis for achieving effective model learning. Step 601 provides necessary data support for the training process by obtaining instance point cloud samples, template point cloud samples, and real point cloud labels. The purpose of this step is to construct a complete and representative training data set so that the shape recovery network can learn the geometric relationship between the instance point cloud and the template point cloud, as well as the matching relationship between the predicted point cloud and the real point cloud.

[0094] Specifically, instance point cloud samples are point cloud fragments generated from multi-view data, which are iteratively merged to form a complete object point cloud, reflecting the geometric characteristics of the object under actual observation conditions. Template point cloud samples are geometric prior representations of each category, which are generated by averaging and optimizing sample point clouds of multiple categories, reflecting the global geometric characteristics and detail distribution of the category. The real point cloud labels are obtained through high-precision 3D scanning equipment, or through manual annotation and verification to ensure their accuracy as reference truth values. These real point cloud labels provide supervision signals for the network, enabling it to effectively correct the prediction results.

[0095] During data acquisition, it is necessary to ensure the diversity and balance of instance point cloud samples and template point cloud samples to cover different shapes and categories of target objects. At the same time, the real point cloud labels need to correspond one-to-one with the instance point cloud and the template point cloud, and be accurately aligned to ensure that the supervisory signal during training can accurately guide network optimization.

[0096] Step 602: Input the instance point cloud samples and the template point cloud samples into the initial network to obtain the point-by-point offset samples and scaling factor samples output by the initial network.

[0097] Specifically, the instance point cloud samples and template point cloud samples are first input into the front-end module of the shape recovery network, including the basic point cloud feature extraction model and the hierarchical point cloud feature extraction model. The basic point cloud feature extraction model extracts the observed geometric features from the instance point cloud samples, and the hierarchical point cloud feature extraction model extracts the category prior geometric features from the template point cloud samples. These features are then fused through the cross-attention mechanism to generate fused features that represent the comprehensive geometric relationship between the instance point cloud and the template point cloud.

[0098] After the feature fusion is completed, the initial network uses these features for further processing to predict the shape recovery parameters of the template point cloud samples, namely the point-by-point offset samples and the scaling factor samples. The point-by-point offset samples are used to represent the adjustment requirements of the template point cloud in the local geometry, and the scaling factor samples are used to represent the adjustment requirements of the template point cloud in the global size. These samples can not only express the geometric relationship between the template point cloud and the instance point cloud, but also provide a quantifiable adjustment basis for the subsequent generation of the predicted point cloud.

[0099] Step 603: transform the template point cloud samples according to the point-by-point offset samples and the scaling factor samples to obtain predicted point cloud samples.

[0100] Specifically, the template point cloud sample is first globally adjusted according to the scaling factor sample. The scaling factor sample provides the global scaling ratio of the template point cloud on the three coordinate axes, and the coordinates of each point of the template point cloud sample are multiplied by the scaling factor of the corresponding axis to generate the scaled point cloud sample. This step realizes the global geometric adjustment of the template point cloud, making its overall size closer to the observed characteristics of the instance point cloud sample.

[0101] Based on the scaled point cloud samples, the point-by-point offset samples are used to further refine the local geometry of the point cloud. The final predicted point cloud samples are generated by adding the coordinates of each point of the scaled point cloud sample to the corresponding point-by-point offset. This point-by-point offset operation can finely correct the differences in the local geometry of the template point cloud, so that the predicted point cloud samples not only maintain the category prior features of the template point cloud, but also fit the observed details of the instance point cloud samples.

[0102] Step 604: Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label.

[0103] Specifically, the second chamfer distance is an indicator that measures the geometric similarity between the predicted point cloud sample and the real point cloud label. Its calculation process includes distance measurements in two directions: on the one hand, for each point in the predicted point cloud sample, find the nearest neighbor point in the real point cloud label, and calculate the square of the Euclidean distance between the two points, and take the average of all these distance squares as the error in the first direction; on the other hand, for each point in the real point cloud label, also find the nearest neighbor point in the predicted point cloud sample, calculate the square of the Euclidean distance between the two points, and take the average as the error in the second direction. Finally, add the errors in the two directions to get the value of the second chamfer distance.

[0104] Through this process, the second chamfer distance can fully reflect the geometric differences between the predicted point cloud samples and the real point cloud labels in three-dimensional space. Due to its bidirectional matching characteristics, this distance can simultaneously evaluate whether the predicted point cloud fully covers the real point cloud and whether there is redundant or unnecessary offset, thereby providing a comprehensive geometric similarity evaluation.

[0105] Step 605: construct a loss function based on the second chamfer distance and the point-by-point offset samples.

[0106] In the training process of the shape recovery network, simply relying on the geometric similarity between the predicted point cloud samples and the real point cloud labels is not enough to ensure the global accuracy and local detail accuracy of the model generation results. In order to further constrain the network output, step 605 constructs a comprehensive loss function by combining the second chamfer distance and the point-by-point offset samples, so that the network can achieve a balance between global consistency and local optimization of shape recovery. The purpose of this step is to provide multi-dimensional supervision signals for network optimization, thereby improving the prediction quality of shape recovery parameters.

[0107] Specifically, the loss function consists of two parts: one is the second chamfer distance, which is used to quantify the geometric difference between the predicted point cloud sample and the real point cloud label; the other is the regularization term of the point-by-point offset sample, which is used to limit the size of the offset to prevent the model from over-adjusting the template point cloud, causing the shape to deviate from the category prior. The second chamfer distance reflects the geometric similarity between the predicted point cloud and the real point cloud through bidirectional nearest neighbor matching, while the regularization term of the point-by-point offset is calculated through the L2 norm, and the squared sum of the offsets of all points is taken as the average as a constraint on the local offset.

[0108] The loss function is expressed by the following formula: ; In the formula, is the loss function, is the real point cloud label, To predict point cloud samples, is the second chamfer distance, is the point-by-point offset sample, is the L2 norm, is the sum of the L2 norm of the point-by-point offset samples, n is the number of points in the predicted point cloud samples, is a hyperparameter.

[0109] In this way, the loss function can constrain the size of the point-by-point offset while optimizing the overall geometric consistency of the predicted point cloud, thereby avoiding overfitting or unreasonable shape adjustment. This comprehensive constraint ensures that the model can match the global characteristics of the real point cloud while maintaining the local rationality of the generated point cloud during shape recovery.

[0110] Step 606: Taking the value of minimizing the loss function as the training goal, the initial network is trained to obtain a shape recovery network.

[0111] Specifically, the training process adopts the supervised learning method, taking instance point cloud samples and template point cloud samples as input, and the real point cloud labels as supervision signals. , using back propagation and gradient descent to optimize network parameters. In each training iteration, instance point cloud samples and template point cloud samples are extracted and fused through the multi-layer feature extraction of the initial network to generate point-by-point offset samples and scaling factor samples, and these parameters are used to transform the template point cloud samples to generate predicted point cloud samples.

[0112] Subsequently, the second chamfer distance is calculated between the predicted point cloud sample and the real point cloud label, and the constraint contribution of each offset sample is calculated through the regularization term. The two parts together constitute the current value of the loss function. The gradient of the loss function relative to the network parameters is calculated through the back-propagation algorithm, and the network weights are updated using an optimization algorithm (such as Adam or SGD) so that the loss function value is gradually reduced.

[0113] During the training process, the network continuously adjusts its weights to better generate predicted point clouds that match the real point cloud labels, while avoiding unreasonable shape adjustments caused by excessive point-by-point offsets. Through multiple rounds of iterative training, the network's prediction ability is gradually improved, and it can eventually generate accurate shape recovery parameters from instance point cloud samples and template point cloud samples.

[0114] Through the implementation of step 606, the shape recovery network optimizes the network weights by minimizing the loss function in multiple rounds of iterations, and successfully trains a model with high-precision shape completion capabilities. The optimized network can take into account both global geometric consistency and local detail correction, thereby achieving accurate shape estimation of dense and messy objects in complex scenes.

[0115] Reference Figure 8 , Figure 8 : is a structural schematic diagram of a shape estimation system for dense and cluttered objects provided by the present invention, the system comprising: An acquisition module is used to acquire an instance point cloud of each object to be evaluated, and to acquire a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to represent the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; A processing module is used to input the instance point cloud and the template point cloud of each object to be evaluated into a shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by the instance point cloud samples and the corresponding real point cloud labels; The transformation module is used to transform the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated.

[0116] In a possible implementation, the acquisition module is further used to acquire color images of an object stack composed of a plurality of objects to be evaluated at a plurality of viewing angles, and to acquire a depth image corresponding to each color image; The processing module is further used to input each color image into a vocabulary segmentation model to obtain an instance segmentation mask and a predicted category of each object to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images under multiple viewing angles; The processing module is further used to combine the instance segmentation mask of each object to be evaluated at each viewing angle with the corresponding depth image to obtain an initial point cloud of each object to be evaluated at each viewing angle; The processing module is also used to perform multiple rounds of iterative merging of initial point clouds with the same prediction category under different viewing angles to obtain instance point clouds of each object to be evaluated.

[0117] In a possible implementation manner, the processing module is further configured to: In each iteration, the first chamfer distance between the initial point with the same predicted category and the currently merged instance point cloud at each different perspective is calculated; When it is determined that the first chamfer distance is less than a preset threshold, merging the initial point cloud with the currently merged instance point cloud; Repeat the above merging process and update the current merged instance point cloud after each round of merging is completed; When it is determined that there is no new initial point cloud that satisfies the condition that the first chamfer distance is less than the preset threshold, the iteration is terminated to obtain the final instance point cloud of each object to be evaluated.

[0118] In a possible implementation manner, the processing module is further configured to: Inputting the instance point cloud of each object to be evaluated into the basic point cloud feature extraction model to obtain the instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic level feature extraction on the input point cloud data; Inputting the template point cloud of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on stratification or regionalization of point cloud data; Through the cross attention mechanism, the fusion features of each object to be evaluated are extracted from the instance point cloud and template point cloud of each object to be evaluated; The template features, fusion features and instance features of each object to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network.

[0119] In a possible implementation manner, the transformation module is further configured to: Multiplying the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain a scaled point cloud of each object to be evaluated; The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0120] In a possible implementation, the system further includes a model training module, which is used to: Obtain instance point cloud samples, template point cloud samples and real point cloud labels; Input the instance point cloud samples and the template point cloud samples into the initial network to obtain the point-by-point offset samples and scaling factor samples output by the initial network; The template point cloud samples are transformed according to the point-by-point offset samples and the scaling factor samples to obtain the predicted point cloud samples; Calculate the second chamfer distance between the predicted point cloud sample and the real point cloud label; Construct a loss function based on the second chamfer distance and the point-by-point offset samples; Taking the value of minimizing the loss function as the training goal, the initial network is trained to obtain the shape recovery network.

[0121] It should be noted that the shape estimation system for dense and cluttered objects provided by the present invention can execute the shape estimation method for dense and cluttered objects of any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0122] Fig. 9 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig. 9As shown, the electronic device may include: a processor 910 (processor), a communication interface 920 (CommunicationsInterface), a memory 930 (memory) and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a shape estimation method for dense and messy objects, the method comprising: obtaining an instance point cloud of each object to be evaluated, and obtaining a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; the instance point cloud and the template point cloud of each object to be evaluated are input into a shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is obtained by training the instance point cloud samples and the corresponding real point cloud labels; the template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameters of each object to be evaluated to obtain the predicted point cloud of each object to be evaluated.

[0123] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the shape estimation method for dense and cluttered objects provided by the above-mentioned embodiments, and the method includes: obtaining an instance point cloud of each object to be evaluated, and obtaining a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; the instance point cloud and the template point cloud of each object to be evaluated are input into a shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by instance point cloud samples and corresponding real point cloud labels; the template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameters of each object to be evaluated to obtain a predicted point cloud for each object to be evaluated.

[0125] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor 910 executes the shape estimation method for dense and cluttered objects provided in the above-mentioned embodiments, the method comprising: obtaining an instance point cloud of each object to be evaluated, and obtaining a template point cloud corresponding to the category of each object to be evaluated; the instance point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; the instance point cloud and the template point cloud of each object to be evaluated are input into a shape recovery network to obtain shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by instance point cloud samples and corresponding real point cloud labels; the template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameters of each object to be evaluated to obtain a predicted point cloud for each object to be evaluated.

[0126] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0127] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A shape estimation method for dense and cluttered objects, characterized in that: include: Obtaining an instance point cloud of each object to be evaluated, and obtaining a template point cloud corresponding to the category of each object to be evaluated; The example point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; Inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape recovery network, and obtaining the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is trained by instance point cloud samples and corresponding real point cloud labels; The template point cloud corresponding to each object to be evaluated is transformed according to the shape recovery parameter of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

2. The shape estimation method for dense and cluttered objects according to claim 1, characterized in that: The step of obtaining the instance point cloud of each object to be evaluated specifically includes: Acquire color images of an object stack composed of a plurality of the objects to be evaluated at multiple viewing angles, and acquire a depth image corresponding to each of the color images; Inputting each of the color images into a vocabulary segmentation model to obtain an instance segmentation mask and a predicted category of each of the objects to be evaluated output by the vocabulary segmentation model; the vocabulary segmentation model is fine-tuned based on sample color images under multiple viewing angles; Combining the instance segmentation mask of each object to be evaluated at each viewing angle with the corresponding depth image to obtain an initial point cloud of each object to be evaluated at each viewing angle; The initial point clouds with the same prediction category under different viewing angles are iteratively merged for multiple rounds to obtain instance point clouds of each object to be evaluated.

3. The shape estimation method for dense and cluttered objects according to claim 2, characterized in that: The step of performing multiple rounds of iterative merging of initial point clouds with the same prediction category under different viewing angles to obtain instance point clouds of each object to be evaluated specifically includes: In each round of iteration, the first chamfer distance between the initial point with the same predicted category and the currently merged instance point cloud under each different viewing angle is calculated; When it is determined that the first chamfer distance is less than a preset threshold, merging the initial point cloud with the currently merged instance point cloud; Repeat the above merging process and update the currently merged instance point cloud after each round of merging is completed; When it is determined that there is no new initial point cloud that satisfies the condition that the first chamfer distance is less than the preset threshold, the iteration is terminated to obtain the final instance point cloud of each object to be evaluated.

4. The shape estimation method for dense and cluttered objects according to claim 1, characterized in that: The shape recovery network includes a basic point cloud feature extraction model and a layered point cloud feature extraction model; Inputting the instance point cloud and the template point cloud of each object to be evaluated into the shape recovery network to obtain the shape recovery parameters of each object to be evaluated output by the shape recovery network specifically includes: Inputting each instance point cloud of the object to be evaluated into the basic point cloud feature extraction model to obtain instance features output by the basic point cloud feature extraction model; the basic point cloud feature extraction model is used to perform global or basic level feature extraction on the input point cloud data; Inputting the template point cloud of each object to be evaluated into the hierarchical point cloud feature extraction model to obtain the template features output by the hierarchical point cloud feature extraction model; the hierarchical point cloud feature extraction model is used to extract multi-level feature representations based on stratification or regionalization of point cloud data; Extracting fusion features of each object to be evaluated from the instance point cloud and the template point cloud of each object to be evaluated through a cross attention mechanism; The template features, the fusion features and the instance features of each of the objects to be evaluated are input into the hierarchical point cloud feature extraction model to obtain the shape restoration parameters of each of the objects to be evaluated output by the shape restoration network.

5. The shape estimation method for dense and cluttered objects according to claim 1, characterized in that: The shape recovery parameters include a scaling factor and a point-by-point offset; the step of transforming the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated specifically includes: Multiplying the scaling factor of each object to be evaluated by the template point cloud corresponding to each object to be evaluated to obtain a scaled point cloud of each object to be evaluated; The scaled point cloud of each object to be evaluated is added to the point-by-point offset of each object to be evaluated to obtain a predicted point cloud of each object to be evaluated.

6. The shape estimation method for dense and cluttered objects according to claim 1, characterized in that: Also included is a training method for the shape recovery network: Obtain instance point cloud samples, template point cloud samples and real point cloud labels; Input the instance point cloud samples and the template point cloud samples into an initial network to obtain point-by-point offset samples and scaling factor samples output by the initial network; Transforming the template point cloud samples according to the point-by-point offset samples and the scaling factor samples to obtain predicted point cloud samples; Calculating a second chamfer distance between the predicted point cloud sample and the real point cloud label; Constructing a loss function according to the second chamfer distance and the point-by-point offset samples; The initial network is trained with minimizing the value of the loss function as a training goal to obtain the shape recovery network.

7. The shape estimation method for dense and cluttered objects according to claim 6, characterized in that: The constructing of a loss function according to the second chamfer distance and the point-by-point offset samples specifically includes: The loss function is expressed by the following formula: ； In the formula, is the loss function, is the real point cloud label, is the predicted point cloud sample, is the second chamfer distance, is the point-by-point offset sample, is the L2 norm, is the sum of the L2 norms of the point-by-point offset samples, n is the number of points in the predicted point cloud samples, is a hyperparameter.

8. A shape estimation system for dense and cluttered objects, characterized in that: include: An acquisition module, used to acquire an instance point cloud of each object to be evaluated, and acquire a template point cloud corresponding to the category of each object to be evaluated; The example point cloud is used to characterize the three-dimensional point cloud data of the object to be evaluated under actual observation conditions, and the template point cloud is used to provide a reference benchmark for completing the three-dimensional point cloud data of the object to be evaluated; A processing module, used for inputting the instance point cloud and the template point cloud of each object to be evaluated into a shape recovery network, and obtaining the shape recovery parameters of each object to be evaluated output by the shape recovery network; the shape recovery network is obtained by training the instance point cloud samples and the corresponding real point cloud labels; The transformation module is used to transform the template point cloud corresponding to each object to be evaluated according to the shape recovery parameters of each object to be evaluated, so as to obtain the predicted point cloud of each object to be evaluated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the shape estimation method for dense and cluttered objects as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the shape estimation method for dense and cluttered objects as claimed in any one of claims 1 to 7 is implemented.

Images