Fragment assembly device and method utilizing point cloud segmentation and registration

Abstract

The present disclosure relates to a fragment assembly device and method utilizing point cloud segmentation and registration. The device comprises: a memory that stores at least one command for fragment assembly utilizing point cloud segmentation and registration; and a processor that performs an operation according to the command, wherein the processor may perform a point cloud segmentation process for dividing into broken pieces by removing an existing area, distinct from an overlapping area that is a broken area, to increase the accuracy of the registration, perform a point cloud registration process for registering the pieces, perform a point cloud XOR process for removing the broken area, and perform a beam search process for detecting inaccurate results.

Description

Fracture assembly device and method using point cloud segmentation and registration

[0001] The present disclosure relates to a fracture assembly device and method, and more particularly, to a fracture assembly device and method that provides a fracture assembly framework based on point cloud segmentation and iterative registration.

[0002] Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application, and their inclusion in this section is not intended to be admitted as prior art.

[0003] Fracture assembly is the process of reassembling the fragments of a broken object to restore its original shape. Figure 1 illustrates an example of fracture assembly (10). Restoring broken fractured pieces to their original shape, such as assembling a jigsaw puzzle or restoring a broken skeleton or statue, is a very difficult task.

[0004] Fracture assembly has attracted attention for tasks such as artifact preservation and 3D object creation, but it remains a challenging task. Previous research on fracture assembly has successfully recovered specific objects, such as cups and jars. However, these methods rely on prior knowledge of shape, limiting their applicability to a wider range of objects.

[0005] Meanwhile, fracture fragments can be broadly categorized into semantic and physical fractures. Semantic fractures involve the removal of meaningful parts from a given object. For example, a chair might be divided into legs, a seat, and a backrest. Physical fractures, on the other hand, occur when an object is shattered by a physical impact. A cup that falls to the floor and shatters into multiple pieces is an example of a physical fracture. These physical fractures present diverse patterns and lack semantic cues, making them a more challenging task than semantic fracture reconstruction.

[0006] When reassembling physically broken pieces, such as a broken cup or plate, the first step is typically to determine which face is the broken face. This is because the broken face is where the fractured piece joins. Next, two pieces with the same broken face are selected and checked to see if they fit. If they do fit, the adjacent pieces are joined together and a new matching face is found. By repeating this process, all broken pieces can eventually be restored to their original state.

[0007] A fracture assembly device and method utilizing point cloud segmentation and registration according to an embodiment provides a new fracture assembly framework based on point cloud segmentation and registration by intuitively utilizing a series of processes performed by humans for fracture assembly.

[0008] A fracture assembly device and method utilizing point cloud segmentation and alignment according to an embodiment can restore a fractured fracture to its original state and reconstruct a crack even when shape and semantic information are insufficient.

[0009] In addition, the fracture assembly device and method according to the embodiment finds the broken part of the fracture by segmenting the point cloud, finds the overlapping part in the refraction area, determines the relative position, and aligns them.

[0010] However, the problems to be solved according to one embodiment are not limited to those mentioned above.

[0011] In order to achieve the above-described technical problem, a fracture assembly device utilizing point cloud segmentation and registration according to the present disclosure comprises: a memory storing at least one command for fracture assembly utilizing point cloud segmentation and registration; and a processor performing an operation according to the command, wherein the processor may perform a point cloud segmentation process for dividing into broken pieces by removing an existing area other than an overlapping area, which is a broken area, to increase the accuracy of the registration, perform a point cloud registration process for joining the pieces, perform a point cloud XOR process for removing the broken area, and perform a beam search process for detecting an inaccurate result.

[0012] At this time, the processor, in the point cloud segmentation process, breaks the fractured segment P b and normal segment P n After dividing and individually segmenting the broken points of each fracture, the segmented results are recorded, the relative poses between adjacent fractures are detected, and the correspondences between fractures are identified through Geotransformer to detect the relative poses between adjacent fractures, outliers for the identified results are filtered, rotation and translation are determined using Singular Value Decomposition (SVD), and the results of the rotation and translation can be refined through Iterative Closest Point (ICP).

[0013] In addition, the processor performs the point cloud XOR process for removing aligned points within a distance smaller than a threshold value during the point cloud matching process, and the point cloud XOR process, since the broken area overlaps only once, minimizes the amount of computation for fracture assembly and optimizes the overlapping ratio between point clouds by removing the overlapping area.

[0014] Additionally, in the above point cloud XOR process, the overlapped points are broken points P b Included in, normal point P n may have been removed during the above point cloud segmentation process.

[0015] Additionally, the processor can search a graph representing a fracture alignment during the beam search process and maintain a set of most promising partial solutions observed in the search.

[0016] Additionally, the processor may select n pairs of fractures with priority given to those having more points, use a matching module to determine relative poses for the n pairs, and, after identifying the poses, merge the fractures through the point cloud XOR process.

[0017] Additionally, the processor may perform a selection process for selecting the top k instances from which the most points have been removed through the point cloud XOR process from the n · k results or the initial n results from the n pairs, and repeat the selection process until a single object remains.

[0018] Additionally, the processor sparsely samples a preset number of points from each fracture surface for segmentation model training and generates a broken point cloud P b or normal point cloud P nLabel the points classified as P on the fractured surface for training the registration model. b is densely sampled and used as input, and the above P b You can use C corresponding to as a label.

[0019] In addition, the fracture assembly method utilizing point cloud segmentation and registration according to the present disclosure may include a step of performing a point cloud segmentation process for dividing a fractured area into broken pieces by removing an existing area that is not an overlapping area, which is a fractured area, to increase the accuracy of registration; a step of performing a point cloud registration process for joining the pieces; a step of performing a point cloud XOR process for removing the fractured area; and a step of performing a beam search process for detecting inaccurate results.

[0020] The fracture assembly device and method utilizing point cloud segmentation and registration according to the present disclosure improve the accuracy of registration by removing unnecessary information in point cloud registration and registration that find the same part of two point clouds and find the relative position.

[0021] In addition, the fracture assembly device and method utilizing point cloud segmentation and alignment according to the present disclosure enable robust restoration of complex fracture fragments.

[0022] In addition, the fracture assembly device and method utilizing point cloud segmentation and alignment according to the present disclosure utilizes point cloud classification for identifying a broken area and point cloud alignment for obtaining relative positions between fracture fragments, thereby enabling the restoration of complex fracture fragments to their original shape.

[0023] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the detailed description of the present invention or the composition of the invention described in the claims.

[0024] Figure 1 is a drawing showing an example of fracture assembly (10).

[0025] FIG. 2 is a drawing showing a process (20) of repeatedly reassembling a rupture into its original shape to improve strength, similar to the way people repair a ruptured part in an embodiment.

[0026] Fig. 3 is a drawing showing an overview (30) of the framework FRASIER according to an embodiment.

[0027] FIG. 4 is a block diagram of a fracture assembly device using a framework according to an embodiment.

[0028] Figure 5 is a drawing showing a qualitative comparison (50) of FRASIER, a framework according to an embodiment, with Global, DGL, and LSTM.

[0029] Figure 6 is a table showing quantitative results (60) of an assembly method according to an embodiment.

[0030] Figure 7 is a diagram showing the results of a removal study (70) for broken point segmentation, point cloud XOR, and beam search.

[0031] Figure 8 is a flowchart showing a fracture assembly process according to an embodiment.

[0032] Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0033] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0034] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0035] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0036] In this specification, the term "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Furthermore, a single unit may be realized using two or more pieces of hardware, and two or more units may be realized by a single piece of hardware.

[0037] Some of the operations or functions described herein as being performed by a terminal, apparatus, or device may instead be performed by a server connected to the terminal, apparatus, or device. Similarly, some of the operations or functions described herein as being performed by a server may also be performed by a terminal, apparatus, or device connected to the server.

[0038] Fracture assembly, or shape assembly, is the process of restoring the original shape of a broken object by assembling the fractures. Fracture assembly encompasses tasks such as assembling puzzles or reassembling broken skeletons or statues. Fracture assembly has attracted attention in computer vision and robotics due to its potential utility in various applications, such as cultural heritage preservation and artifact conservation, but it continues to be a challenging task.

[0039] Depending on their cause, ruptures can be broadly classified into two types: semantic ruptures and physical ruptures. Specifically, rupture segmentation can decompose a given object into a set of segments based on their semantic meaning. Such semantic ruptures can be efficiently reassembled utilizing both semantic and geometric cues. On the other hand, physical impacts can be applied to a given object, which leads to more natural and diverse rupture patterns. Physical ruptures are more challenging than semantic ruptures due to the diverse and unexpected patterns and the lack of semantic cues.

[0040] If we were to imagine a situation where we were trying to reassemble physically broken (shattered) fractures, we would examine the set of broken (shattered) fractures, imagining at a high level which fractures are geometrically adjacent. We would then select two fractures that seem to fit well together and assemble them into a single fracture by matching their adjacent sides. If these two fractures were originally a single piece, we would treat them as a single piece and continue by selecting another fracture adjacent to the newly integrated piece. By repeating the above process, we can finally assemble the broken (shattered) fractures into a single whole. The above sequence of steps corresponds to the intuitive way people generally reassemble fractures.

[0041] FIG. 2 is a drawing showing a process (20) of repeatedly reassembling a rupture into its original shape to improve strength, similar to the way people repair a ruptured part in an embodiment.

[0042] Inspired by the intuitive human approach, the present invention provides a novel framework for fracture assembly, as illustrated in Fig. 2. Specifically, the present invention utilizes point cloud registration, which aligns two point clouds captured from different perspectives. However, fracture assembly differs from typical point cloud registration because fractures exhibit extreme conditions compared to scene registration data with sufficient overlap. To bridge this gap, the present invention proposes a fracture-specific registration approach to effectively locate and align fractured areas.

[0043] Furthermore, the embodiment provides a point cloud XOR concept to remove unnecessary points in the fractured area after registration. Overall, the framework provided in the embodiment recognizes the fractured area of ​​each fracture through point cloud segmentation, and then iteratively aligns two feasible fractures by finding various paths through beam search and trial and error. Furthermore, the embodiment evaluates the effectiveness of the proposed framework using the Breaking Bad dataset. The framework provided in the embodiment demonstrates robustness in fracture assembly tasks and significantly outperforms previous state-of-the-art techniques. Furthermore, the embodiment thoroughly analyzes each part of the framework to better understand its influence on the assembly results.

[0044] Hereinafter, operations related to the framework of a fracture assembly device according to an embodiment are described.

[0045] Automatic Part Assembly

[0046] To automatically reconstruct 3D objects, numerous approaches have been developed for reassembly. Some focus on semantic information about fractures from fracture patterns, accurately positioning them globally. These methods are generally effective in reconstructing categorized objects with segmentable parts, such as chairs, desks, and cars. In contrast, other methods focus on geometrically segmented fractures that lack semantic cues or exhibit randomness. To address the irregularity of fractures, these methods employ methods such as reconstruction priorities or local feature matching. However, these methods struggle with robustness when dealing with complexly fractured objects of diverse categories. In this work, we address this challenge by utilizing robust registration methods and enhance robustness through efficient point cloud removal and search methods.

[0047] Point Cloud Registration

[0048] Point cloud registration, which aligns two point clouds with a certain degree of overlap, can be applied to determine the relative positions of adjacent fractures. Correspondence-based approaches first identify correspondences between points in the two point clouds, which serve as constraints guiding the alignment. Advances in data-driven approaches have enabled the process to be completed end-to-end, with point features derived through ML algorithms or directly obtained through correspondences. In this embodiment, GeoTransformer is used for fracture alignment due to its robust rotational tolerance and robust performance in low-overlap scenarios.

[0049] Point cloud segmentation

[0050] 3D point cloud segmentation can be applied to a variety of scenarios where point clusters exhibit shared essential or semantic features. This diversity is enabled by the permutation and translation invariance implemented in the model architecture. PointNet introduced permutation invariance to handle irregular point shapes through max pooling, and subsequent models leveraged features in local regions in a manner similar to convolutional neural networks in 2D image processing. In particular, PointNext demonstrates excellent performance and scalability, acting as a semantic cue for ruptured surfaces, enabling our method to leverage semantic information even in complex rupture scenarios.

[0051] Given a rupture set F = {P i} in P i represents a point cloud uniformly sampled from the i-th rupture surface, and the rupture assembly is P i Rotation of R i ∈ SO(3) and transformation t i ∈ R 3 The goal is to reconstruct the original object shape by recovering the original object shape. This problem can be considered a special case of the 3D registration problem. In particular, if two adjacent fractures sharing a common surface in F can be distinguished, this assembly problem can be redefined as a set of iterative registration problems.

[0052] P i Wow P j Assuming that P are two ruptures sharing a common surface, i Wow P j The assembly of can be a matching problem of finding relative positions for potential correspondences, as shown in Equation 1.

[0053]

[0054] In mathematical expression 1, (p, q) is P i Wow P jrepresents the potential correspondence between points, and C represents the set of potential correspondences. However, unlike general registration problems where there is a certain overlap between point clouds, the overlap between ruptures is extremely limited, making rupture assembly a challenging task. Furthermore, because ruptures are randomly distributed, the connectivity between ruptures is uncertain, requiring the identification of neighboring pairs. Furthermore, ruptured surfaces typically have fewer unique features than scene registration data, leading to inaccurate alignment of ruptures.

[0055] Below, we describe the FRASIER framework according to an embodiment inspired by human intuitive behavior for fracture assembly.

[0056] Figure 3 is a schematic diagram (30) of a framework (FRASIER) according to an embodiment. Referring to Figure 3, when sampled points for each rupture are provided, robust part assembly is performed using point cloud segmentation. This allows for the transformation of difficult part assembly problems into classical point cloud registration. In the embodiment, the point cloud is visualized as a shape for better understanding.

[0057] Specifically, in this embodiment, points in the fracture surface area of ​​each fracture are first classified to enable alignment. Pairwise point cloud alignment is then iteratively performed on the fracture rupture points. Specifically, points in adjacent areas are excluded when merging two fractures into a unified fracture. Furthermore, beam search is applied within this fracture assembly framework to ensure robustness against our pairwise alignment strategy.

[0058] A fracture assembly device and method utilizing point cloud segmentation and registration according to an embodiment provides a new fracture assembly framework based on point cloud segmentation and registration using a human intuitive method.

[0059] A fracture assembly device and method utilizing point cloud segmentation and alignment according to an embodiment can restore a fractured fracture to its original state and reconstruct a crack even when shape and semantic information are insufficient.

[0060] In addition, the fracture assembly device and method according to the embodiment finds the broken part of the fracture by segmenting the point cloud, finds the overlapping part in the refraction area, determines the relative position, and aligns them.

[0061] FIG. 4 is a block diagram of a fracture assembly device using a framework according to an embodiment.

[0062] The fracture assembly device (100) configuration illustrated in Fig. 4 is merely a simplified example.

[0063] The communication module (110) can be configured regardless of the communication mode, such as wired or wireless, and can be configured with various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the communication module (110) can operate based on the well-known World Wide Web (WWW), and can also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth. For example, the communication module (110) can be responsible for transmitting and receiving data required to perform a technique according to an embodiment of the present disclosure.

[0064] The memory (120) may refer to any type of storage medium. For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk. Such a memory (120) may also constitute a database as illustrated in FIG. 1.

[0065] The memory (120) can store at least one instruction that can be executed by the processor (130). In addition, the memory (120) can store any type of information generated or determined by the processor (130) and any type of information received by the server (200). For example, the memory (120) stores RM data and RM protocols according to the user, as will be described later. In addition, the memory (120) stores various types of modules, instruction sets, and models.

[0066] The processor (130) may perform technical features according to embodiments of the present disclosure, which will be described later, by executing at least one instruction stored in the memory (120). In one embodiment, the processor (130) may be configured with at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of a computer device.

[0067] Below, the fracture assembly operation of the processor (130) utilizing point cloud segmentation and alignment is described.

[0068] The processor (130) first performs a broken point segmentation process and then performs a point cloud registration process (Broken Point Segmentation and Registration) to join the pieces.

[0069] During the assembly process, the damaged area occupies a small portion of the total surface area. The low overlap condition poses a challenge to the typical registration method optimized for scenes with a high overlap ratio. To address this issue, the embodiment accurately identifies the damaged area and removes unnecessary areas, thereby enabling more accurate registration. To this end, the embodiment uses point cloud segmentation to distinguish between damaged and normal areas. Specifically, the embodiment uses Pointnext, which can capture both small local details and large-scale structures using multi-scale features, for point cloud segmentation. Based on Pointnext, each rupture P is divided into a damaged segment P. b and normal segment P n is identified as . At this time, P = P b ∪ P nIn the embodiment, this broken point splitting is performed for each rupture before alignment.

[0070] Afterwards, the processor (130) individually segments the broken points of each fracture, and the next step is to find the relative positions between adjacent fractures. For this purpose, the embodiment uses a correspondence-based registration method based on Geotransformer, which is robust in overlapping scenarios and allows for rigid transformation. First, the processor (130) identifies the correspondences between fractures through Geotransformer, and then applies RANSAC to remove outliers. Afterwards, the singular value decomposition (SVD) is used to determine the rotation and translation, and the rotation and translation results are further refined through Iterative Closest Point (ICP).

[0071] After registration, the processor performs a point cloud XOR process. After registration, it is important to combine the identified correct fracture pairs. Therefore, the point clouds must be merged. The most basic approach is a union operation on a set of point clouds. However, the union operation expands the size of the point cloud as iterations progress, increasing the computational cost for subsequent registration operations. One way to prevent this is to subsample after the union. However, this reduces the registration accuracy as the point cloud becomes coarser. To solve this problem, the processor (130) according to the embodiment performs a point cloud XOR. The point cloud XOR proposed in the embodiment is a process of removing points aligned within a distance smaller than a threshold. The point cloud XOR process is derived from the observation that overlapping areas can be removed because the damaged areas overlap only once. This point cloud XOR method not only minimizes the amount of computation and computational cost, but also optimizes the overlap ratio between point clouds. The overlapping points are broken points P. b belongs to, normal point P n are removed in advance in point cloud segmentation. That is, overlapping points are normal points P n The broken point P has been removed in advance from this point cloud segmentation. b Belongs to.

[0072] Afterwards, the processor (130) performs a beam search process. Although the splitting and XOR modules significantly improve the overlap ratio between the broken parts, the matching result is not always accurate. The important point is that a single inaccurate result can adversely affect the overall reconstruction in the iterative process. To prevent this, in the embodiment, the processor (130) uses beam search for various path exploration. Beam search is a search algorithm that maintains the set of most promising partial solutions observed so far while exploring the rupture-sorted graph. The details of the algorithm are described below.

[0073] First, the processor (130) selects n pairs of ruptures, prioritizing those with more points. A matching module is used to determine the relative poses for these n pairs. In an embodiment, after identifying the poses, the processor (130) merges the ruptures using point cloud XOR. At this time, the top k instances with the most points removed are selected from among the n pairs or initially n results. This process is repeated until a single object remains.

[0074] In an embodiment, the processor (130) sparsely samples a preset number of points (e.g., 5,000) from each surface of the fracture for segmentation model training, and generates a fractured point cloud P b or normal point cloud P n Labeling is performed on the classified ones. During training, the AdamW optimizer is used to compute the cross-entropy loss between the estimated and actual values. Additionally, for evaluation, the point cloud is sampled proportionally to the surface area of ​​each rupture.

[0075] In the embodiment, for training the matching model, in the embodiment, point P in the damaged part of the fracture surface bDensely sample and use C corresponding to the sampling result as a label. For inference, the estimated broken point P in the segmentation module b Use .

[0076] Additionally, in the embodiment, n = 6 and k = 3 are set in the beam search. In other words, in the embodiment, six pairs are selected from the largest area ruptures and their relative poses are evaluated through the matching module. If 3 × 6 results or six results are initially given, the top three results can be selected and the rest can be discarded.

[0077] Below, the experimental process of a fracture assembly device utilizing point cloud segmentation and alignment according to an embodiment is described.

[0078] In this example, the Breaking Bad dataset, consisting of three categories, is used for experiments. Among the three categories, artifact category data is used for training, while artifact and everyday data are used for evaluation.

[0079] Additionally, metrics used in the evaluation include the root mean square error (RMSE) for rotation and translation and part accuracy (PA). Part accuracy (PA) is the percentage of perfectly positioned parts based on the chamfer distance.

[0080] In this example, we evaluate the performance of our framework (Frasier) against four baseline methods in the context of a 3D fracture assembly task. All methods were trained on artifact category data. Figure 5 provides a qualitative comparison (50) of our framework, FRASIER, with Global, DGL, and LSTM. Compared to other methods, our approach demonstrates superior performance on both simple fractures (A) and complex fractures (B). The qualitative results presented in Figure 5 provide a gallery of reconstructions from our framework and baseline approaches. While other baselines struggle with simple fractures, our fracture assembly method can reconstruct even complex fractures. Figure 6 is a table showing quantitative results (60) of our assembly method according to the example. Referring to Figure 6, FRASIER outperforms previous methods in RMSE (R), RMSE (T), and PA. Ours(unseen) represents the results evaluated on unseen category data, where (R) and (T) represent rotation and translation, respectively.

[0081] As shown in Fig. 6, the fracture assembly method according to the embodiment significantly outperforms existing methods, achieving an improvement of more than 30 degrees in the RMSE(R) metric compared to Jigsaw. Furthermore, the last row of the table in Fig. 6 includes the evaluation results on unseen data. In the embodiment, we evaluated the model on everyday data to evaluate its generalization performance. Even when tested on an unseen dataset, our method performs similarly to the evaluation using the same categories for training. This demonstrates that the fracture assembly method according to the embodiment is robust to distribution shifts, which are essential for real-world applications. The visualization of Jigsaw is omitted from Fig. 5 due to the lack of open-source code, and the table in Fig. 6 directly cites quantitative results and the results obtained in Table 2 of Jiaxin Lu, Yifan Sun, and Qixing Huang, "Jigsaw: Learning to assemble multiple fractured objects," arXiv preprint arXiv:2305.17975, 2023.

[0082] In addition, the embodiment investigated the influence of individual modules on the reconstruction quality. Fig. 7 is a diagram showing the results of an ablation study (70) for broken point segmentation, point cloud XOR, and beam search. As shown in Fig. 7, the results from simple registration to FRASIER are compared by adding each module with the results of ablation of each module. In Fig. 7, Beam represents beam search, Seg represents point cloud segmentation, and XOR represents point cloud XOR.

[0083] The first image (Registration Only) in Figure 7 is an unsatisfactory result, with all components omitted. The rupture often overlaps the volume or floats in mid-air. However, with the addition of three components, the reconstruction becomes more realistic. In particular, the segmentation of the broken points and point cloud XOR significantly improve the assembly process by removing unnecessary points from the broken area.

[0084] In this paper, we present an innovative framework inspired by the human assembly process. Utilizing point cloud segmentation and registration methods, the framework enables the reassembly of complex fractures back to their original form. Furthermore, experimental results demonstrate that point cloud XOR and beam exploration significantly improve registration quality. Finally, FRASIER significantly outperforms state-of-the-art techniques and demonstrates robustness when evaluating unseen data during training.

[0085] Below, a fracture assembly method utilizing point cloud segmentation and registration is sequentially described. Since the function of the fracture assembly method according to the embodiment is essentially the same as the function of the fracture assembly device utilizing point cloud segmentation and registration, any description overlapping with FIGS. 2 to 7 will be omitted.

[0086] Figure 8 is a flowchart illustrating a fracture assembly process using point cloud segmentation and registration according to an embodiment.

[0087] Referring to FIG. 8, in step S100, a point cloud segmentation step is performed to divide the point cloud into broken pieces by removing existing areas that are not overlapping areas, which are broken areas, to increase the accuracy of the matching. In step S200, a point cloud registration step is performed to join the pieces. In step S300, a point cloud XOR step is performed to remove the broken areas, and in step S400, a beam search step is performed to detect inaccurate results.

[0088] A fracture assembly device and method utilizing point cloud segmentation and registration according to an embodiment improves the accuracy of registration by removing unnecessary information from point cloud registration and registration that find the same part of two point clouds and find the relative position.

[0089] In addition, the fracture assembly device and method utilizing point cloud segmentation and alignment according to the embodiment enable robust restoration of complex fracture fragments.

[0090] In addition, the fracture assembly device and method utilizing point cloud segmentation and alignment according to the embodiment utilizes point cloud classification to identify a broken area and point cloud alignment to obtain relative positions between fracture fragments, thereby enabling the restoration of complex fracture fragments to their original shape.

[0091] A model in this specification may refer to any form of computer program that operates based on a network function, an artificial neural network, and / or a neural network. Throughout this specification, the terms model, neural network, network function, and neural network may be used interchangeably. A neural network is a network in which one or more nodes are interconnected through one or more links to form input node and output node relationships within the neural network. The characteristics of a neural network can be determined based on the number of nodes and links within the neural network, the correlation between the nodes and links, and the weight value assigned to each link. A neural network may be composed of a set of one or more nodes. A subset of the nodes constituting the neural network may constitute a layer.

[0092] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. A deep neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a generative adversarial network (GAN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, a generative adversarial network (GAN), a transformer, and the like. The description of the above-described deep neural network is merely an example, and the present disclosure is not limited thereto.

[0093] Neural networks can learn through at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, self-supervised learning, or reinforcement learning. Neural network learning can be the process of applying knowledge to the neural network to perform a specific action.

[0094] Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, labeled data is used for each training data, while unsupervised learning uses unlabeled data. The amount of change in the connection weights of each updated node can be determined by the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate can vary depending on the number of iterations in the neural network's training cycle. Additionally, to prevent overfitting, methods such as increasing the learning data, regularization, dropout that disables some nodes, and batch normalization layers can be applied.

[0095] In one embodiment, the model may borrow at least a portion of a transformer. The transformer may be composed of an encoder that encodes embedded data and a decoder that decodes the encoded data. The transformer may have a structure that receives a series of data and outputs a series of data of different types through encoding and decoding steps. In one embodiment, the series of data may be processed into a form operable by the transformer. The process of processing the series of data into a form operable by the transformer may include an embedding process. Expressions such as data tokens, embedding vectors, and embedding tokens may refer to data embedded in a form operable by the transformer.

[0096] To encode and decode a series of data, a transformer can utilize an attention algorithm to process the encoders and decoders within the transformer. An attention algorithm can refer to an algorithm that, for a given query, calculates the similarity for one or more keys, reflects this similarity in the values ​​corresponding to each key, and then weights and sums the values ​​to which the similarity is reflected to calculate an attention value.

[0097] Depending on how the query, key, and value are configured, various types of attention algorithms can be categorized. For example, if attention is obtained by setting the query, key, and value all to the same value, this could be a self-attention algorithm. If attention is obtained by reducing the dimensionality of the embedding vector to process a series of input data in parallel and then generating individual attention heads for each segmented embedding vector, this could be a multi-head attention algorithm.

[0098] In one embodiment, the transformer may be composed of modules that perform multiple multi-head self-attention algorithms or multi-head encoder-decoder algorithms. In one embodiment, the transformer may also include additional components other than attention algorithms, such as embedding, normalization, and softmax. Methods for constructing a transformer using attention algorithms may include methods disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

[0099] A transformer can be applied to various data domains, such as embedded natural language, segmented image data, and audio waveforms, to transform a series of input data into a series of output data. To transform data with various data domains into a series of data that can be input to a transformer, the transformer can embed the data. The transformer can process additional data that expresses the relative positional relationship or phase relationship between the series of input data. Alternatively, vectors expressing the relative positional relationship or phase relationship between the input data can be additionally reflected in the series of input data to embed the series of input data. In one example, the relative positional relationship between the series of input data may include, but is not limited to, word order within a natural language sentence, the relative positional relationship between each segmented image, and the time order of segmented audio waveforms. The process of adding information expressing the relative positional relationship or phase relationship between the series of input data may be referred to as positional encoding.

[0100] In one embodiment, the model may include, but is not limited to, at least one of a Recurrent Neural Network (RNN), a Long Short Term Memory (LSTM) network, a Deep Neural Network (DNN), a Convolutional Neural Network (CNN), and a Bidirectional Recurrent Deep Neural Network (BRDNN).

[0101] In one embodiment, the model may be a model trained using transfer learning. Transfer learning, in this context, refers to a learning method that pre-trains a large amount of unlabeled training data using semi-supervised or self-learning methods to obtain a pre-trained model for a first task, then fine-tunes the pre-trained model to suit a second task, and trains it on labeled training data using supervised learning to implement a target model.

[0102] The disclosed content is merely an example, and various modifications and implementations can be made by a person skilled in the art without departing from the gist of the claims claimed in the patent, so the scope of protection of the disclosed content is not limited to the specific embodiments described above.

Claims

1. A memory storing at least one command for fracture assembly using point cloud segmentation and registration; and A processor comprising: a processor that performs an operation according to the above command; The above processor, In order to increase the accuracy of the above alignment, a point cloud segmentation process is performed to divide the point cloud into broken pieces by removing the existing area, which is not the overlapping area, which is the broken area. Perform a point cloud registration process to join the above pieces, Perform a point cloud XOR process to remove the above broken areas, A fracture assembly device that performs a beam search process to detect inaccurate results.

2. In the first paragraph, the processor, In the above point cloud segmentation process, the broken segment P of the fracture b and normal segment P n Divide it, After individually segmenting the broken points of each fracture, record the segmented results. Detect the relative pose between adjacent fractures, To detect the relative positions between the adjacent fractures, the correspondence between the fractures is identified through Geotransformer, Filter out outliers from the above identified results, Use singular value decomposition (SVD) to determine rotations and translations, A fracture assembly device that refines the results of the above rotation and translation through an ICP (Iterative Closest Point).

3. In paragraph 1, The above processor performs the point cloud XOR process for removing aligned points within a distance smaller than a threshold value during the point cloud matching process, The above point cloud XOR process is a fracture assembly device that minimizes the amount of computation for fracture assembly and optimizes the overlapping ratio between point clouds by removing the overlapping area since the broken area overlaps only once.

4. In the third paragraph, in the point cloud XOR process, Overlapped points are broken points P b Included in, Normal point P n A fracture assembly device that is removed during the above point cloud segmentation process.

5. In the first paragraph, the processor, In the above beam search process, the graph showing the fracture alignment is searched, A fracture assembly device that maintains the most promising partial solution set observed in the above exploration.

6. In paragraph 5, the processor, Select n pairs of fractures with preference to those with more points, To determine the relative posture for the above n pairs, we use a matching module, A fracture assembly device that merges fractures through the point cloud XOR process after identifying the above posture.

7. In the 6th paragraph, the processor, In the above n pairs, a selection process is performed to select the top k instances from which the most points have been removed through the point cloud XOR process, from the n · k results or the initial n results, A fracture assembly device that repeats the above selection process until a single object remains.

8. In the first paragraph, the processor, To train the segmentation model, a preset number of points are sparsely sampled from each fracture surface, Broken Point Cloud P b or normal point cloud P n Label the points classified as For training the registration model, the P on the fractured surface of the fracture b is densely sampled and used as input, and the above P b A fracture assembly device using a C corresponding to the label.

9. In a fracture assembly method performed by a processor of a device, A step of performing a point cloud segmentation process to divide the point cloud into broken pieces by removing existing areas that are not overlapping areas, which are broken areas, to increase the accuracy of alignment; A step of performing a point cloud registration process for joining the above pieces; A step of performing a point cloud XOR process to remove the above broken area; and A fracture assembly method utilizing point cloud segmentation and alignment, comprising a step of performing a beam search process to detect inaccurate results.

10. In the 9th paragraph, the processor, In the above point cloud segmentation process, the broken segment P of the fracture b and normal segment P n Divide it, After individually segmenting the broken points of each fracture, record the segmented results. Detect the relative pose between adjacent fractures, To detect the relative positions between the adjacent fractures, the correspondence between the fractures is identified through Geotransformer, Filter out outliers from the above identified results, Use singular value decomposition (SVD) to determine rotations and translations, A fracture assembly method utilizing point cloud segmentation and registration, refining the results of the above rotation and translation through ICP (Iterative Closest Point).

11. In paragraph 9, The above processor performs the point cloud XOR process for removing aligned points within a distance smaller than a threshold value during the point cloud matching process, The above point cloud XOR process minimizes the amount of computation for fracture assembly by removing the overlapping area, since the broken area overlaps only once, and optimizes the overlapping ratio between point clouds. In the above point cloud XOR process, the overlapped points are broken points P b Included in, normal point P n A fracture assembly method utilizing point cloud segmentation and alignment, which is removed in the above point cloud segmentation process.

12. In paragraph 9, the processor, In the above beam search process, the graph showing the fracture alignment is searched, A fracture assembly method utilizing point cloud segmentation and registration that retains the most promising partial solution set observed in the above exploration.

13. In the 12th paragraph, the processor, Select n pairs of fractures with preference to those with more points, To determine the relative posture for the above n pairs, we use a matching module, A fracture assembly method utilizing point cloud segmentation and alignment, which identifies the above-mentioned posture and then merges the fractures through the point cloud XOR process.

14. In the 13th paragraph, the processor, In the above n pairs, a selection process is performed to select the top k instances from which the most points have been removed through the point cloud XOR process, from the n · k results or the initial n results, A fracture assembly method utilizing point cloud segmentation and alignment, repeating the above selection process until a single object remains.

15. In paragraph 9, the processor, To train the segmentation model, a preset number of points are sparsely sampled from each fracture surface, Broken Point Cloud P b or normal point cloud P n Label the points classified as For training the registration model, the P on the fractured surface of the fracture b is densely sampled and used as input, and the above P b A fracture assembly method utilizing point cloud segmentation and alignment, using C corresponding to the label.

Images