Three-dimensional garment segmentation method and device

Abstract

The invention is suitable for the technical field of point cloud data processing, and provides a three-dimensional clothing segmentation method and device. For a to-be-segmented target needing to be segmented, initial point cloud data and multi-view image data of the to-be-segmented target are obtained and preprocessed, and then feature extraction and cross-modal feature fusion are performed on the point cloud data and the target multi-view image data obtained through preprocessing so as to enhance the segmentation precision. The method comprises the steps of extracting features of different resolutions by using multi-scale convolution, splicing the features, classifying the spliced fusion features of each point through a full connection layer FC, determining a segmentation label of each point in the point cloud data, and finally segmenting the initial point cloud data through the segmentation labels. The problems that an existing three-dimensional garment segmentation technology lacks the processing capacity for complex texture and fine-grained garment features and is high in calculation cost are solved.

Description

Technical Field

[0001] The present application relates to the technical field of point cloud data processing, and in particular to a three-dimensional clothing segmentation method and device. Background Art

[0002] Three-dimensional clothing segmentation has made significant progress in recent years. The existing technologies mainly focus on how to efficiently handle complex clothing shapes and details. MGN-Seg is an optimization-driven method that achieves three-dimensional clothing segmentation through SMPL+D registration, but it is expensive and can only handle a limited number of clothing categories. GIM3D provides a rich three-dimensional clothing dataset by using clothing models generated from synthetic data, and supports segmentation tasks on meshes and point clouds. CloSe-Net is the latest technology that specializes in processing color point clouds and uses prior knowledge between clothing and the human body to significantly improve the performance of fine-grained segmentation.

[0003] Although the existing 3D clothing segmentation technology has made significant progress, it still has some objective defects. MGN-Seg relies on SMPL+D registration and multi-view rendering. The processing process is time-consuming and computationally expensive. Each scan takes 15-20 minutes, and it is limited to three types of clothing, with poor scalability. Although GIM3D has solved some of the data shortage problems through synthetic datasets, the segmentation categories are still limited to three categories (tops, bottoms, and human bodies), and it lacks the ability to process complex textures and fine-grained clothing features. Although CloSe-Net has improved segmentation performance, it relies on point cloud data and still faces challenges in dealing with complex clothing shapes and material changes. Summary of the invention

[0004] In view of this, the present application provides a three-dimensional clothing segmentation method and device to solve the problems that the existing three-dimensional clothing segmentation technology lacks the ability to process complex textures and fine-grained clothing features and has high computational cost.

[0005] In a first aspect, the present application provides a three-dimensional clothing segmentation method, which is applied to a segmentation model. The method comprises: Get the point cloud data of the target to be segmented and multi-view image data , and through the point cloud data and the multi-view image data determines The point cloud data The two-dimensional projection point of ; Extract the point cloud data Geometric features And the multi-view image data Texture shape features , and combined with the two-dimensional projection point Perform cross-modal feature fusion to obtain fusion features ; The multi-scale features of the fusion features at different resolutions are extracted by multi-scale convolution , and then splice the multi-scale features at different scale levels to obtain the spliced ​​multi-scale features ,in, Indicates the scale level of the convolutional layer; Through the fully connected layer The multi-scale features after splicing Classification is performed to determine a segmentation label for each point in the point cloud data, and the initial point cloud data is segmented according to the segmentation label using the segmentation model.

[0006] Optionally, the fully connected layer The multi-scale features after splicing The classification includes: By formula Determine the probability that each point in the point cloud data belongs to each segmentation label , and then through the probability Determine the segmentation label for each point.

[0007] Optionally, the probability of passing Determining the segmentation label for each point includes: The probability of each point The highest segmentation label is determined as the segmentation label for the point.

[0008] Optionally, the method further includes: By formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

[0009] Optionally, the method further includes: Get the target point cloud data R of the click area through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

[0010] A second aspect of the present application provides a three-dimensional clothing segmentation device, which is applied to a segmentation model, and the device comprises: Target data acquisition unit, used to obtain point cloud data of the target to be segmented and multi-view image data , and through the point cloud data and the multi-view image data determines The point cloud data The two-dimensional projection point of ; A first feature determination unit is used to extract the point cloud data Geometric features And the multi-view image data Texture shape features , and combined with the two-dimensional projection point Perform cross-modal feature fusion to obtain fusion features ; The second feature determination unit is used to extract multi-scale features of the fusion feature at different resolutions through multi-scale convolution. , and then splice the multi-scale features at different scale levels to obtain the spliced ​​multi-scale features ,in, Indicates the scale level of the convolutional layer; Data segmentation unit, used to pass through the fully connected layer The multi-scale features after splicing Classification is performed to determine a segmentation label for each point in the point cloud data, and the initial point cloud data is segmented according to the segmentation label using the segmentation model.

[0011] Optionally, the data segmentation unit passes through a fully connected layer The multi-scale features after splicing The classification includes: By formula Determine the probability that each point in the point cloud data belongs to each segmentation label , and then through the probability Determine the segmentation label for each point.

[0012] Optionally, the probability of Determining the segmentation label for each point includes: The probability of each point The highest segmentation label is determined as the segmentation label for the point.

[0013] Optionally, the device further comprises: The loss function is determined by the formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

[0014] Optionally, the device further comprises: The second segmentation unit is used to obtain the target point cloud data R of the click area through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

[0015] In the embodiment provided in the present application, for the target to be segmented, its initial point cloud data and multi-view image data are first obtained and preprocessed, and then feature extraction and cross-modal feature fusion are performed on the preprocessed point cloud data and target multi-view image data to enhance the segmentation accuracy. Multi-scale convolution is then used to extract features of different resolutions and splice them. The fused features of each point after splicing are classified to determine the segmentation label of each point in the point cloud data, and finally the initial point cloud data is segmented by the segmentation label.

[0016] The advantages of this application are: 1) Cross-modal feature fusion improves segmentation accuracy: This application makes full use of 3D geometric information and texture details of 2D images by cross-modal feature fusion of point cloud and image. Compared with the method that only uses geometric information, this fusion can significantly improve the accuracy of fine-grained segmentation, especially when dealing with complex clothing structures and details.

[0017] 2) Multi-scale feature learning enhances the processing ability of details and the whole: This application can capture the global outline and local details of clothing at the same time. This multi-scale feature extraction mechanism makes up for the shortcomings of traditional single-scale methods in distinguishing different levels of clothing structure, making the segmentation results more refined.

[0018] 3) Lightweight interactive optimization improves practicality and flexibility: This application uses a lightweight interactive optimization module to allow users to make local adjustments when errors occur in the segmentation results without having to recalculate the global results. This local optimization strategy significantly reduces the consumption of computing resources and can quickly respond to user feedback, improving the practicality of the system and user experience.

[0019] 4) Improved computational efficiency: Through point cloud downsampling, local optimization strategies, and effective feature fusion methods, this application improves segmentation accuracy while reducing computational costs. Compared with traditional methods that require long-term processing of complex pipelines (such as MGN-Seg, etc.), this solution has higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of the method provided in the embodiment of the present application; Figure 2 The overall framework diagram of the solution provided for the embodiment of the present application; Figure 3 A structural diagram of a device provided in an embodiment of the present application; Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0022] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0023] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0024] The present application provides a three-dimensional clothing segmentation method and device to solve the problems that the existing three-dimensional clothing segmentation technology lacks the ability to process complex textures and fine-grained clothing features and has high computational cost.

[0025] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0026] like Figure 1As shown, it is a flow chart of a three-dimensional clothing segmentation method provided by the present application. The method is applied to the segmentation model. The steps of the process include: S101, obtaining point cloud data and multi-view image data of the target to be segmented , and through the point cloud data and the multi-view image data determines The point cloud data The two-dimensional projection point of .

[0027] In this embodiment, the initial point cloud data of the target to be segmented, such as clothing, can be obtained through a three-dimensional scanning device or an existing data set. These data contain the three-dimensional coordinates (x, y, z) and color information (r, g, b) of the clothing. In order to reduce the amount of calculation, the initial point cloud data can be downsampled by a preset method, such as voxel grid filtering, random sampling consistency, spherical neighborhood search, etc., to ensure that the main geometric features are retained while reducing the number of points. Thus, point cloud data is obtained. . This data describes the geometry and color of the garment.

[0028] For multi-view image data , the clothing can be photographed from different angles, and the obtained two-dimensional clothing images are determined as multi-view image data , which can provide additional texture and detail information.

[0029] Get point cloud data and multi-view image data Then, the point cloud data is projected onto each view image based on the calibrated geometric projection model to obtain the point cloud data. The two-dimensional projection point of The specific formula is , is the preset projection matrix. The projected point Corresponding to the pixel points on the two-dimensional image, it is convenient for subsequent feature fusion.

[0030] S102, extracting the point cloud data Geometric features And the multi-view image data Texture shape features , and combined with the two-dimensional projection point Perform cross-modal feature fusion to obtain fusion features .

[0031] In this embodiment, for point cloud feature extraction, since Point Transformer is based on the self-attention mechanism, it can adaptively capture local and global geometric information by dynamically calculating the relative relationship between points. Therefore, the Point Transformer network can be used to extract point cloud data. Perform feature extraction to obtain the geometric features of each point , which is represented by its characteristics: This feature contains spatial context information and is particularly suitable for clothing with complex geometric shapes.

[0032] For image feature extraction, the pre-trained convolutional neural network Swin Transformer can be used to extract features from each view image. Extract texture and shape features, expressed as The image features It can capture the subtle texture and pattern information of the clothing surface, which is especially important for fine-grained segmentation.

[0033] After the point cloud features and image features are extracted, the point cloud projected onto the image is combined with the corresponding image features to complete the feature fusion of the point cloud and the image. , get image features And with point cloud features To perform the fusion: , thereby obtaining the fusion feature ,This fusion feature combines three-dimensional geometric information with two-dimensional texture and color information,,which can enhance the segmentation accuracy.

[0034] This embodiment makes full use of the 3D geometric information and the texture details of the 2D image by fusing the cross-modal features of the point cloud and the image. Compared with the method that only uses geometric information, this fusion can significantly improve the accuracy of fine-grained segmentation, especially when dealing with complex clothing structures and details.

[0035] S103, extracting multi-scale features of the fused features at different resolutions through multi-scale convolution ,in, Indicates the scale level of the convolutional layer.

[0036] In this embodiment, in order to cope with the complexity of clothing at different levels, this embodiment uses multiple convolution layers with different convolution kernel sizes to learn fusion features from different spatial resolutions. The specific process is as follows: 1. Multi-scale feature calculation.

[0037] For each fused point , use multi-scale convolution to extract features of different resolutions: .in, Represents the scale level of the convolution layer. Through convolution kernels of different scales, the network can focus on the global clothing contour and local detail structure respectively.

[0038] 2. Combine multi-scale features.

[0039] By formula This multi-scale learning mechanism enables the model to capture features of different scales when dealing with fine-grained clothing segmentation, which helps to improve the precision of segmentation.

[0040] S104, through the fully connected layer The multi-scale features after splicing Classification is performed to determine a segmentation label for each point in the point cloud data, and the initial point cloud data is segmented according to the segmentation label using the segmentation model.

[0041] In this embodiment, several different clothing labels are pre-set as segmentation labels, such as segmentation label A sleeve, segmentation label B interlayer, segmentation label C pocket, etc. Then through the fully connected layer Classify the multi-scale features after splicing each point, and realize the formula Point cloud data Each point in the segmentation label prediction is performed, and the prediction results are Determine the segmentation label for each point. The prediction results is {segmentation label A: 1.44, segmentation label B: 1.01, segmentation label C: 0.67...}. If the segmentation label with the closest prediction result to 1 is set as the segmentation label of the point, then the point The segmentation label is B sandwich. Finally determine the point cloud data After the segmentation labels of each point in the point cloud are obtained, the segmentation model can be used to segment the point cloud data. In another embodiment, the probability of each point can also be The highest segmentation label is determined as the segmentation label for the point.

[0042] In another embodiment, the method further comprises: By formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

[0043] Taking the segmentation labels including ABC as an example, C represents the number of different segmentation labels, which is 3. N is the point cloud data The total number of all points in .

[0044] In this embodiment, the point cloud data needs to be A true label is preset for each point in the image. It can be set according to the region. For example, the target to be segmented is pre-divided into multiple regions, and a clothing label is set as the true label for each region. The loss function uses the standard cross entropy loss function. After updating the segmentation model through this loss function, the segmentation accuracy of clothing can be improved.

[0045] For example, the segmentation labels of clothing are shown in the following table.

[0046] Serial number Split Label 1 T-shirt 2 Shirt 3 Vest 4 Coat 5 Jacket 6 Hoodies 7 Pants 8 Short-Pants 9 Skirts 10 Dress 11 Scarf 12 Hat 13 Shoes 14 Body 15 Hair Our method is compared with DGCNN, DeltaConv and CloSe-Net methods. The results are shown in the following table. The table shows the intersection over union (IoU) of each class and the average value (IoUmean) of all classes. It can be seen that the proposed method has the highest segmentation accuracy.

[0047] Method Mean T-shirt Shirt Vest Coat Jacket Hoodies Pants DGCNN 86.21 87.88 81.02 90.58 81.6 96.16 97.46 82.5 DeltaConv 83.37 87.22 73.56 84.68 80.06 98.52 96.99 78.37 CloSe-Net 90.98 95.47 92.94 98.86 90.12 99.23 99.43 85.96 Ours 91.38 95.99 93.32 98.7 90.48 99.37 99.26 85.79 In another embodiment, the method further comprises: Get the target point cloud data R of the click area through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

[0048] In this embodiment, in order to improve the practicality and flexibility of the segmentation model, a lightweight interactive optimization module is designed to allow users to provide feedback by clicking on the error area when the segmentation result is wrong. When the user clicks on an area, the segmentation model can determine a local loss function for the clicked area, and then perform segmentation using the segmentation model updated by the local loss function. The specific segmentation method and segmentation point cloud data The method is the same as that of , which will not be described here. In this embodiment, it is not necessary to recalculate the entire segmentation result when segmenting the local area. This lightweight interactive optimization greatly reduces the consumption of computing resources and enables the segmentation result to quickly respond to the user's correction requirements.

[0049] So far, completed Figure 1 The process shown.

[0050] The overall framework diagram of the above steps S101-S104 of this application is as follows Figure 2As shown, the technical effects achieved by this application are as follows: 1. Cross-modal feature fusion: Use point cloud and multi-view image data for feature fusion, combine the geometric information of 3D point cloud with the texture and color features of 2D image, and project 3D points into 2D image through geometric projection model. The fusion method proposed in this scheme improves segmentation accuracy and overcomes the limitations of single modality processing.

[0051] 2. Multi-scale feature learning: A multi-scale convolution module is designed to learn and fuse features from different resolutions. Through convolution kernels of different scales, the network can capture the global structure of clothing while focusing on local details, effectively improving the fine-grained segmentation capability.

[0052] 3. Lightweight interactive optimization: A lightweight local optimization mechanism based on user feedback is proposed. Users can make local adjustments by clicking on the error area without recalculating the entire model. This module greatly reduces the consumption of computing resources while improving the flexibility and responsiveness of the model.

[0053] In the embodiment of the present application, for the target to be segmented, its initial point cloud data and multi-view image data are first obtained and preprocessed, and then feature extraction and cross-modal feature fusion are performed on the preprocessed point cloud data and target multi-view image data to enhance the segmentation accuracy. Multi-scale convolution is then used to extract features of different resolutions and splice them. The fused features of each point after splicing are classified to determine the segmentation label of each point in the point cloud data, and finally the initial point cloud data is segmented by the segmentation label. This solves the problem that the existing 3D clothing segmentation technology lacks the ability to process complex textures and fine-grained clothing features and has high computational cost.

[0054] The present application also provides a three-dimensional clothing segmentation device, which is applied to the segmentation model, such as Figure 3 As shown, the device comprises: The target data acquisition unit 301 is used to acquire the point cloud data of the target to be segmented. and multi-view image data , and through the point cloud data and the multi-view image data determines The point cloud data The two-dimensional projection point of ; The first feature determination unit 302 is used to extract the point cloud data Geometric features And the multi-view image data Texture shape features , and combined with the two-dimensional projection point Perform cross-modal feature fusion to obtain fusion features ; The second feature determination unit 303 is used to extract multi-scale features of the fusion feature at different resolutions through multi-scale convolution. , and then splice the multi-scale features at different scale levels to obtain the spliced ​​multi-scale features ,in, Indicates the scale level of the convolutional layer; The data segmentation unit 304 is used to pass through the fully connected layer The multi-scale features after splicing Classification is performed to determine a segmentation label for each point in the point cloud data, and the initial point cloud data is segmented according to the segmentation label using the segmentation model.

[0055] In another embodiment, the data segmentation unit passes through a fully connected layer The multi-scale features after splicing The classification includes: By formula Determine the probability that each point in the point cloud data belongs to each segmentation label , and then through the probability Determine the segmentation label for each point.

[0056] In another embodiment, the data segmentation unit passes the probability Determining the segmentation label for each point includes: The probability of each point The highest segmentation label is determined as the segmentation label for the point.

[0057] In another embodiment, the apparatus further comprises: The loss function determination unit 305 is used to determine the loss function by formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

[0058] In another embodiment, the apparatus further comprises: The second segmentation unit 306 is used to obtain the target point cloud data R of the click area, through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

[0059] The above embodiment of the present invention provides a three-dimensional clothing segmentation method, and based on the method provides a three-dimensional clothing segmentation device. Through the above method and device, the problems of the existing three-dimensional clothing segmentation technology lacking the ability to process complex textures and fine-grained clothing features and high computational cost can be solved.

[0060] This embodiment also discloses a computer device, such as Figure 4 As shown, the computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement any of the above-mentioned three-dimensional clothing segmentation methods.

[0061] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A three-dimensional clothing segmentation method, characterized in that: Applied to the segmentation model, the method comprises: Acquire point cloud data and multi-view image data of the target to be segmented, and determine the two-dimensional projection points of the point cloud data through the point cloud data and the multi-view image data; Extracting geometric features of the point cloud data and texture shape features of the multi-view image data, and performing cross-modal feature fusion in combination with the two-dimensional projection points to obtain fused features; Extracting multi-scale features of the fused features at different resolutions through multi-scale convolution, and then splicing the multi-scale features at different scale levels to obtain spliced ​​multi-scale features; The spliced ​​multi-scale features are classified through a fully connected layer FC to determine a segmentation label for each point in the point cloud data, and the initial point cloud data is segmented according to the segmentation label through the segmentation model.

2. The method according to claim 1, characterized in that The classifying the spliced ​​multi-scale features through the fully connected layer FC includes: By formula Determine the probability that each point in the point cloud data belongs to each segmentation label , and then through the probability Determine the segmentation label for each point, where is the multi-scale feature after splicing.

3. The method according to claim 2, characterized in that The probability of passing Determining the segmentation label for each point includes: The probability of each point The highest segmentation label is determined as the segmentation label for the point.

4. The method according to claim 1, characterized in that: The method further comprises: By formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

5. The method according to claim 4, characterized in that The method further comprises: Get the target point cloud data R of the click area through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

6. A three-dimensional clothing segmentation device, characterized in that: Applied to the segmentation model, the device comprises: A target data acquisition unit, used to acquire point cloud data and multi-view image data of the target to be segmented, and determine a two-dimensional projection point of the point cloud data through the point cloud data and the multi-view image data; A first feature determination unit, configured to extract geometric features of the point cloud data and texture shape features of the multi-view image data, and perform cross-modal feature fusion in combination with the two-dimensional projection points to obtain fused features; A second feature determination unit is used to extract multi-scale features of the fused features at different resolutions through multi-scale convolution, and then splice the multi-scale features at different scale levels to obtain spliced ​​multi-scale features; The data segmentation unit is used to classify the spliced ​​multi-scale features through the fully connected layer FC, determine the segmentation label of each point in the point cloud data, and segment the initial point cloud data according to the segmentation label through the segmentation model.

7. The device according to claim 6, characterized in that The data segmentation unit classifies the spliced ​​multi-scale features through the fully connected layer FC, including: By formula Determine the probability that each point in the point cloud data belongs to each segmentation label , and then through the probability Determine the segmentation label for each point, where is the multi-scale feature after splicing.

8. The device according to claim 7, characterized in that The probability of passing through the data segmentation unit Determining the segmentation label for each point includes: The probability of each point The highest segmentation label is determined as the segmentation label for the point.

9. The device according to claim 6, characterized in that The device also includes: The loss function is determined by the formula Determine the loss function of the segmentation model and update the segmentation model by the loss function, where C is the number of different segmentation labels and N is the point cloud data The total number of points, Yes The default true label, is the predicted probability of the segmentation model.

10. The device according to claim 9, characterized in that The device also includes: The second segmentation unit is used to obtain the target point cloud data R of the click area through the formula A local loss function of the click area is determined, and the target point cloud data R is segmented using a segmentation model updated by the local loss function.

Images