A furniture image marking method, model training method and device

Abstract

The embodiment of the present application provides a method for labeling furniture images, a model training method and an apparatus. In the embodiment of the present application, the component labels to which each spatial point in the 3D furniture model belongs can be determined based on point cloud data; the 3D furniture model can be converted into a 2D furniture image, and the component labels that have been marked in the 3D furniture model can also be mapped to the corresponding pixel points in the 2D furniture image, and the 2D furniture image can be recolored based on the coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. Accordingly, the 2D furniture image, component segmentation map and component labels corresponding to the 3D furniture model can be used as training samples to train the furniture component segmentation model, so that the furniture component segmentation model learns the knowledge of component segmentation in the 2D furniture image, and thus, the trained furniture component segmentation model can be used to realize automatic component-level labeling of massive two-dimensional furniture images.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a furniture image marking method, model training method and equipment. Background Art

[0002] For scenarios such as furniture e-commerce or interior design, it is necessary to provide furniture product filtering options. Currently, it is usually necessary to manually fill in the component attributes of furniture products as the basis for filtering options.

[0003] However, due to labor cost constraints, filling efficiency is usually low, and this results in a large number of furniture products lacking information on the attributes of these components, resulting in insufficient filtering results under the filter items, affecting the experience of merchants and users. Summary of the Invention

[0004] Various aspects of the present application provide a furniture image marking method, a model training method, and an apparatus for automatically marking 2D furniture images at the component level.

[0005] The present application provides a method for training a furniture parts segmentation model, including:

[0006] Obtain point cloud data corresponding to the 3D furniture model;

[0007] Based on the point cloud data, determining the component labels of the spatial points included in the 3D furniture model;

[0008] Converting the 3D furniture model into a 2D furniture image;

[0009] Under the 2D furniture image, based on the mapping relationship between pixel points and spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels, the 2D furniture image is recolored to generate a component segmentation map corresponding to the 3D furniture model;

[0010] The 2D furniture images, component segmentation maps, and component labels corresponding to the 3D furniture models are used as training samples to train a furniture component segmentation model.

[0011] The present application also provides a method for marking a furniture image, including:

[0012] Obtain a two-dimensional furniture image corresponding to the target furniture;

[0013] Inputting the two-dimensional furniture image into a furniture component segmentation model, wherein the furniture component segmentation model is trained using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples, and the component segmentation map is obtained by recoloring the 2D furniture image based on the mapping relationship between pixels in the 2D furniture image and spatial points of the 3D furniture model, the component labels to which the spatial points belong, and coloring parameters configured for different component labels;

[0014] In the furniture component segmentation model, image segmentation is performed on the two-dimensional furniture image to generate a component segmentation map corresponding to the two-dimensional furniture image;

[0015] Wherein, each segmented area in the component segmentation diagram is marked with a component label.

[0016] An embodiment of the present application further provides a computing device, including a memory and a processor;

[0017] The memory is used to store one or more computer instructions;

[0018] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0019] Obtain point cloud data corresponding to the 3D furniture model;

[0020] Based on the point cloud data, determining the component labels of the spatial points included in the 3D furniture model;

[0021] Converting the 3D furniture model into a 2D furniture image;

[0022] Under the 2D furniture image, based on the mapping relationship between pixel points and spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels, the 2D furniture image is recolored to generate a component segmentation map corresponding to the 3D furniture model;

[0023] The 2D furniture images, component segmentation maps, and component labels corresponding to the 3D furniture models are used as training samples to train a furniture component segmentation model.

[0024] An embodiment of the present application further provides a computing device, including a memory and a processor;

[0025] The memory is used to store one or more computer instructions;

[0026] The processor is coupled to the memory and configured to execute the one or more computer instructions for:

[0027] Obtain a two-dimensional furniture image corresponding to the target furniture;

[0028] Inputting the two-dimensional furniture image into a furniture component segmentation model, wherein the furniture component segmentation model is trained using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples, and the component segmentation map is obtained by recoloring the 2D furniture image based on the mapping relationship between pixels in the 2D furniture image and spatial points of the 3D furniture model, the component labels to which the spatial points belong, and coloring parameters configured for different component labels;

[0029] In the furniture component segmentation model, image segmentation is performed on the two-dimensional furniture image to generate a component segmentation map corresponding to the two-dimensional furniture image;

[0030] Wherein, each segmented area in the component segmentation diagram is marked with a component label.

[0031] In an embodiment of the present application, starting from a 3D furniture model, based on point cloud data, the component labels belonging to each spatial point in the 3D furniture model can be determined; on this basis, the 3D furniture model can be converted into a 2D furniture image, and the component labels already marked in the 3D furniture model can also be mapped to the corresponding pixels in the 2D furniture image. The 2D furniture image can then be recolored based on the coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. In this way, the component labels determined under the 3D furniture model can be presented in the 2D furniture image in the form of color. Accordingly, for the same 3D furniture model, a 2D furniture image, a component segmentation map, and component labels can be obtained. After these are input as training samples into the furniture component segmentation model, the furniture component segmentation model can learn the knowledge of component segmentation in 2D furniture images. Thus, the trained furniture component segmentation model can be used to annotate component labels on any two-dimensional furniture image, thereby achieving automatic component-level labeling for massive furniture images, saving a lot of manpower and material resources, and overcoming the labeling problem in furniture scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0033] Figure 1 A flowchart of a method for training a furniture component segmentation model provided by an exemplary embodiment of the present application;

[0034] Figure 2 A logical diagram of a training scheme for a furniture parts segmentation model provided by an exemplary embodiment of the present application;

[0035] Figure 3 A flowchart of a method for marking a furniture image provided by another exemplary embodiment of the present application is provided;

[0036] Figure 4 A comparison diagram before and after marking provided for another exemplary embodiment of the present application;

[0037] Figure 5 A schematic structural diagram of a computing device is provided as another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0038] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0039] At present, in furniture-related scenarios, it is usually necessary to rely on manual labor to fill in component attributes, and the filling efficiency and completeness are both poor. To this end, in some embodiments of the present application: starting from the 3D furniture model, based on the point cloud data, the component labels belonging to each spatial point in the 3D furniture model can be determined; on this basis, the 3D furniture model can be converted into a 2D furniture image, and the component labels that have been marked in the 3D furniture model can also be mapped to the corresponding pixel points in the 2D furniture image, and the 2D furniture image can be recolored based on the coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. In this way, the component labels determined under the 3D furniture model can be presented in the 2D furniture image in the form of color. Based on this, for the same 3D furniture model, 2D furniture images, component segmentation maps and component labels can be obtained. After using these as training samples to input into the furniture component segmentation model, the furniture component segmentation model can learn the knowledge of component segmentation in 2D furniture images. Therefore, the trained furniture component segmentation model can be used to annotate component labels on any two-dimensional furniture image, thereby realizing automatic component-level labeling of massive furniture images, saving a lot of manpower and material resources, and overcoming the labeling difficulties in furniture scenes.

[0040] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0041] Figure 1 A flowchart of a method for training a furniture component segmentation model provided as an exemplary embodiment of the present application. Figure 2This is a logical diagram of a training scheme for a furniture parts segmentation model provided by an exemplary embodiment of the present application. The method can be executed by a data processing device, which can be implemented as a combination of software and / or hardware, and the data processing device can be integrated into a computing device. Figure 1 As shown, the method may include:

[0042] Step 100: Obtain point cloud data corresponding to the 3D furniture model;

[0043] Step 101: Based on the point cloud data, determine the component labels of each spatial point included in the 3D furniture model;

[0044] Step 102: Convert the 3D furniture model into a 2D furniture image;

[0045] Step 103: Recolor the 2D furniture image based on the mapping relationship between pixels and spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels, to generate a component segmentation map corresponding to the 3D furniture model.

[0046] Step 104: Train a furniture component segmentation model using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples.

[0047] The training scheme of the furniture component segmentation model provided in this embodiment can be applied to various scenarios that require furniture component segmentation, such as home decoration design, furniture e-commerce, etc. This embodiment does not limit the application scenarios.

[0048] The following is a brief explanation of some technical terms mentioned in this embodiment.

[0049] Components: These can refer to the main functional parts of furniture. For example, for a chair, its components may include legs, backrest, armrests, load-bearing surface, etc.; for a bed, its components may include: sleeping area, headboard, bed frame, etc.

[0050] Component label: It can be a label used to describe component-level attributes. In this embodiment, the component label may include but is not limited to a component name label, a component texture label, a component shape label, or a component material label, etc. The component label can be customized according to actual needs.

[0051] Point cloud data: This refers to a dataset of spatial points of a 3D furniture model in a world coordinate system. In one alternative, the 3D furniture model's triangle mesh data, texture data, and metadata such as point clouds, vertices, and face indices can be acquired and used to construct point cloud data. Point cloud data contains multidimensional data describing the spatial points of the 3D furniture model in a world coordinate system.

[0052] 2D furniture image: This refers to a two-dimensional image of furniture from a certain perspective. For example, a furniture image with a white or solid color background and the furniture itself as the main subject.

[0053] On this basis, reference Figure 1 For ease of description, in this embodiment, the training scheme of the furniture component segmentation model will be explained from the perspective of a 3D furniture model. It should be understood that in actual applications, a large number of training samples can be obtained after processing a large number of 3D furniture models, thereby continuously optimizing the furniture component segmentation model.

[0054] refer to Figure 1 In step 100, point cloud data corresponding to the 3D furniture model can be obtained. The 3D furniture model in this embodiment can be sourced from a large-scale 3D furniture model library open sourced by the industry. For example, the datasets ModelNet40, ShapeNet, 3D-FRONT, etc., which are publicly available in the academic industry, are not limited to this in this embodiment. These model libraries contain at least the metadata of the 3D furniture models mentioned above. In this embodiment, point cloud data of the 3D furniture model can be constructed based on these metadata. Of course, in some model libraries, point cloud data has been pre-configured. In this embodiment, the point cloud data provided by such model libraries can be directly used.

[0055] In step 101, the component labels associated with each spatial point in the 3D furniture model can be determined based on the point cloud data. As previously mentioned, component labels can be customized as needed. Furthermore, the point cloud data contains multidimensional data describing the spatial points of the 3D furniture model in the world coordinate system. Therefore, the point cloud data can reflect the component attribute status of each spatial point in the 3D furniture model. Thus, the component labels associated with each spatial point in the 3D furniture model can be determined by analyzing the point cloud data and the attribute characteristics required by each component label.

[0056] In step 102, the 3D furniture model can also be converted into a 2D furniture image according to the point cloud data. In this process, the 3D furniture model can be rasterized according to the point cloud data to map the spatial points in the world coordinate system contained in the 3D furniture model into pixel points in the perspective coordinate system to obtain a two-dimensional image; according to the texture data of the triangular mesh where the spatial points contained in the 3D furniture model are located, the corresponding pixel points in the two-dimensional image are rendered to generate a 2D furniture image corresponding to the 3D furniture model. In the rasterization process, the rasterizer can control the cameras to map the spatial points in the world coordinate system contained in the 3D furniture model into pixel points in the perspective coordinate system through a series of affine transformations, projection transformations, etc. The perspective coordinate system here can be an orthogonal perspective coordinate system, a perspective perspective coordinate system or other custom perspective coordinate systems. Different perspective coordinate systems can convert the 3D furniture model into a 2D furniture image under different perspectives. In step 102, the rasterization operation can be used to determine the triangular meshes that intersect with each pixel point in the two-dimensional image in the triangular mesh contained in the 3D furniture model. The triangular mesh is drawn based on the spatial point. Therefore, a mapping relationship between the pixel points in the two-dimensional image and the spatial point in the 3D furniture model can be further generated. It is worth noting that in 3D objects, geometric transformation, geometric detection, animation, rendering and coloring operations are usually performed based on triangular meshes. For this reason, in this embodiment, in the process of rendering the two-dimensional image obtained after the rasterization process, the pixel value of the pixel point can actually be calculated based on the texture data of the triangular mesh that intersects with each pixel point in the two-dimensional image, thereby achieving rendering. It can be seen that in this embodiment, the rendered 2D furniture image can be considered as a real two-dimensional image of the furniture, for example, a furniture white background image.

[0057] Based on this, in this embodiment, a 2D furniture image can be rendered based on the point cloud data, triangulated mesh data, and texture data corresponding to the 3D furniture model. Furthermore, during the rendering process, rendering conditions such as lighting and viewing angle can be configured to more realistically restore the 2D furniture image corresponding to the 3D furniture model. In this embodiment, the 2D furniture image corresponding to the 3D furniture model is retained for future use.

[0058] In step 103, the 2D furniture image can also be recolored based on the mapping relationship between pixel points and spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. An exemplary component segmentation map can be referenced. Figure 2As shown. The mapping relationship between pixel points and spatial points can be generated in the aforementioned step 102, and the component labels to which the spatial points belong can be generated in step 101. In addition, in this embodiment, coloring parameters can be configured for each component label. Optionally, different coloring parameters can be configured for different component labels to better distinguish different component labels. Of course, this embodiment is not limited to this, and other color schemes can also be used to distinguish component labels. For example, the coloring parameters of component labels can be configured with the goal of different colors for adjacent components.

[0059] Taking the first pixel in a 2D furniture image as an example, during the recoloring process: based on the mapping relationship between pixels and spatial points, the target spatial point associated with the first pixel in the 2D furniture image can be determined; based on the component label to which the target spatial point belongs, the target component label corresponding to the first pixel can be determined; based on the coloring parameters configured for the target component label, the pixel value of the first pixel can be calculated; and based on the pixel value of the first pixel, the first pixel can be recolored according to the pixel value of the first pixel; wherein the first pixel is any pixel in the 2D furniture image. In this way, the component label associated with the target spatial point associated with the first pixel can be presented in the form of color at the corresponding position in the 2D furniture image; and from the perspective of the dimensionality of the 2D furniture image, the component label can be presented in the form of color on different components.

[0060] At this point, the 2D furniture image and component segmentation map corresponding to the 3D furniture model can be obtained. The visual effect comparison between the 2D furniture image and component segmentation map corresponding to the 3D furniture model can be referred to Figure 2 In addition, through the above processing, 2D furniture images and component segmentation maps can be generated for a large number of 3D furniture models, thereby providing a large number of training samples for the furniture component segmentation model.

[0061] On this basis, in step 104, the 2D furniture images, component segmentation maps, and component labels corresponding to the 3D furniture model can be used as training samples to train the furniture component segmentation model. In this embodiment, the input of the furniture component segmentation model can be configured as a 2D furniture image, and the output can be configured as a component segmentation map and component labels. In this way, after the 2D furniture images and component segmentation maps corresponding to the 3D furniture model are input into the furniture component segmentation model as training samples, the furniture component segmentation model can extract features from the 2D furniture images and use the corresponding component segmentation maps as supervision and component labels to optimize the model parameters of the furniture component segmentation model, thereby learning the knowledge of component segmentation of the 2D furniture images. Among them, the furniture component segmentation model can use an image segmentation network such as HRnet. Of course, this embodiment is not limited to this, and CNN, RNN, etc. can also be used. Taking HRnet as an example, more competitive high-level semantic features can be obtained by parallel connection of high-level semantic features and low-level semantic features and repeated fusion of feature representations generated from high-level to low-level layers. This overcomes the problem of feature information loss in the training process and generates component labels through multi-scale and multi-dimensional segmentation. Moreover, through the reasonable design of the upsampling process, good accuracy can be obtained and computational complexity can be reduced.

[0062] As explained above, in this embodiment, a large number of training samples can be obtained after processing a large number of 3D furniture models, thereby continuously optimizing the furniture component segmentation model. In this regard, during the training process of the furniture component segmentation network, the training samples can also be divided into two categories: training samples and verification samples. For example, 80% of all training samples can be used as training samples, and the remaining 20% ​​can be used as verification samples. In this way, the furniture component segmentation model can be trained using training samples, while the furniture component segmentation model can be tested using verification samples. If the accuracy rate obtained in the test is greater than the preset accuracy rate or the accuracy rate converges within a certain period of time, the training can be terminated. Otherwise, more training samples can be used to continue model training.

[0063] In addition, in this embodiment, data enhancement can be performed on the training samples to increase the order of magnitude of the training samples. Data enhancement methods may include, but are not limited to, normalization, random flipping, random cropping, random scaling, brightness increase, edge enhancement, and other operations. It should be understood that other furniture images obtained after data enhancement for a certain 2D furniture image have component labels consistent with the 2D furniture image, except that the presentation position, area size, etc. of the same component label in different images may differ. Accordingly, by enhancing the diversity of data distribution, the overfitting phenomenon encountered during the training process can be suppressed, thereby improving the accuracy and generalization of the furniture component segmentation model.

[0064] Based on this, in this embodiment, starting from a 3D furniture model, based on point cloud data, the component labels belonging to each spatial point in the 3D furniture model can be determined. Based on this, the 3D furniture model can be converted into a 2D furniture image, and the component labels already labeled in the 3D furniture model can be mapped to the corresponding pixels in the 2D furniture image. The 2D furniture image can then be recolored based on coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. In this way, the component labels determined in the 3D furniture model can be presented in the 2D furniture image in the form of color. Thus, for the same 3D furniture model, a 2D furniture image, a component segmentation map, and component labels can be obtained. These can be used as training samples to input into a furniture component segmentation model, allowing the furniture component segmentation model to learn how to segment components in 2D furniture images. Consequently, the trained furniture component segmentation model can be used to annotate component labels on any 2D furniture image, thereby enabling automatic component-level labeling of massive furniture images, saving significant manpower and resources and overcoming the labeling challenges in furniture scenes.

[0065] In the above or following embodiments, the point cloud data corresponding to the 3D furniture model may be input into the point cloud segmentation model, and the point cloud segmentation model may be used to respectively determine the component labels to which each spatial point included in the 3D furniture model belongs.

[0066] In this embodiment, in the point cloud segmentation model, the various spatial points contained in the 3D furniture model can be clustered to obtain at least one spatial point set; based on the point cloud data, attribute feature extraction is performed under at least one spatial point set to obtain attribute features corresponding to at least one spatial point set; based on the mapping relationship between the attribute features and the component labels and the attribute features corresponding to at least one spatial point set, the component label to which the at least one spatial point set belongs is output.

[0067] When clustering the spatial points in a 3D furniture model, clustering can be done based on their coordinates and texture data. This allows points with similar textures to be grouped into the same set of points, which aligns perfectly with the requirements for component segmentation. Therefore, clustering allows the spatial points in a 3D furniture model to be grouped into sets of points that match the components. In other words, different sets of points correspond to different components, thus overcoming the problem of point cloud data disorder.

[0068] Afterwards, component labels can be annotated based on spatial point sets. The mapping relationship between the attribute features of spatial point sets and component labels is pre-learned in the point cloud segmentation model. Therefore, in this embodiment, the point cloud segmentation model can first extract attribute features from at least one spatial point set. In the process of attribute feature extraction, a deep learning network such as CNN can be used to map the point cloud data corresponding to the spatial point set into a high-level feature space to characterize the attributes of the spatial point set through attribute features, and a neural network for solving classification problems can be used to classify the spatial point set based on the attribute features, and the spatial point set can be classified under a certain component label. Accordingly, in this embodiment, the input of the point cloud segmentation model can be configured as point cloud data, and the output can be configured as multiple preset component labels. In this way, after the point cloud data corresponding to the 3D furniture model is input into the point cloud segmentation model, the point cloud segmentation model can output the component label to which each spatial point set is classified, thereby obtaining the component label of each spatial point contained in the 3D furniture model, wherein the component label of each spatial point in the same spatial point set is consistent.

[0069] The following describes the training process of the point cloud segmentation network:

[0070] In this embodiment, a number of labeled 3D furniture samples can be obtained, and the 3D furniture samples contain point cloud data and component labels belonging to each spatial point in the 3D model; the number of labeled 3D furniture samples are input into a point cloud segmentation model; in the point cloud segmentation model, the spatial points in the 3D model are clustered and attribute features are extracted; based on the extracted attribute features and the component labels belonging to each spatial point, the mapping relationship between the attribute features and the component labels is learned.

[0071] Currently, labeled 3D furniture samples can come from manual labeling or from existing model libraries. It should be understood that the number of labeled 3D furniture samples is typically relatively small. In this embodiment, a point cloud segmentation network can be trained based on a small number of labeled 3D furniture samples. This point cloud segmentation network can then be used to annotate component labels for a large number of 3D furniture models, thereby supporting the generation of the aforementioned massive training samples for training the furniture component segmentation model.

[0072] During the training process, a supervised training method is used, and the component labels of each spatial point in the annotated 3D model are used as supervision to train the point cloud segmentation network. During the training process of the point cloud segmentation network, the labeled 3D furniture samples can also be divided into two categories: training samples and verification samples. For example, 80% of all labeled 3D furniture samples can be used as training samples, and the remaining 20% ​​can be used as verification samples. In this way, the point cloud segmentation model can be trained using training samples, and the point cloud segmentation model can be tested using verification samples. If the accuracy obtained from the test is greater than the preset accuracy or the accuracy converges within a certain period of time, the training can be terminated. Otherwise, more labeled 3D furniture samples can be used to continue model training.

[0073] Furthermore, in this embodiment, data enhancement can be performed on the labeled 3D furniture samples to increase the number of labeled 3D furniture samples. Data enhancement methods may include, but are not limited to, normalization, random flipping, and random stretching. It should be understood that after data enhancement of the point cloud data of a particular 3D model, other 3D models obtained will have component labels consistent with that 3D model, with the difference being the presentation position and spatial dimensions of the same component labels in different 3D models. Thus, by increasing the diversity of the data distribution, overfitting during training can be suppressed, thereby improving the accuracy and generalization of the point cloud segmentation model.

[0074] Based on this, in this embodiment, a point cloud segmentation network can be trained based on a small number of labeled 3D furniture samples. On this basis, simply inputting the point cloud data corresponding to the 3D furniture model into the point cloud segmentation network can obtain the component labels belonging to each spatial point in the 3D furniture model. Consequently, the point cloud segmentation network can be used to automatically label component labels for massive amounts of 3D furniture models, providing a data foundation for training samples for the furniture component segmentation model.

[0075] Figure 3 This is a flowchart of a method for marking furniture images provided by another exemplary embodiment of the present application. The method can be executed by a data processing device, which can be implemented as a combination of software and / or hardware, and the data processing device can be integrated into a computing device. Figure 3 , the method may include:

[0076] Step 300: Acquire a two-dimensional furniture image corresponding to the target furniture;

[0077] Step 302: Input the 2D furniture image into a furniture component segmentation model. The furniture component segmentation model is trained using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples. The component segmentation map is obtained by recoloring the 2D furniture image based on the mapping relationship between pixels in the 2D furniture image and spatial points in the 3D furniture model, the component labels to which the spatial points belong, and coloring parameters configured for different component labels.

[0078] Step 303: In the furniture component segmentation model, perform image segmentation on the two-dimensional furniture image to generate a component segmentation map corresponding to the two-dimensional furniture image;

[0079] Each segmented area in the component segmentation diagram is marked with a component label.

[0080] The furniture image marking method provided in this embodiment can be applied to scenarios where component-level marking of two-dimensional furniture images is performed, such as furniture e-commerce, interior design, etc. This embodiment does not limit the application scenarios.

[0081] Taking the furniture e-commerce scenario as an example, based on the furniture image labeling solution provided in this embodiment, massive furniture images can be automatically labeled at the component level, and the labeled furniture images can be uploaded to the e-commerce platform as product display images. In this way, the massive furniture images on the e-commerce platform can be associated with complete component labels, and these component labels can be used as screening information in stages such as product push and product search, thereby better supporting the e-commerce platform's processing of furniture products.

[0082] Regarding the training process of the furniture component segmentation model, reference may be made to the descriptions in the aforementioned embodiments related to the training method of the furniture component segmentation model, which will not be described in detail here.

[0083] In this embodiment, a 2D furniture image corresponding to the target furniture can be input into the furniture component segmentation model. Based on the learned image segmentation knowledge, the furniture component segmentation model can segment and recolor the 2D furniture image to output a component segmentation map. Each component in the component segmentation map can be associated with a component label. Figure 4 A comparison diagram before and after marking is provided for another exemplary embodiment of the present application. Figure 4 The 2D furniture image corresponding to the target furniture can be a 2D image that truly represents the target furniture, such as a white background image. In the component segmentation map, each segmented area is recolored and labeled with the corresponding component label. This allows the component segmentation map to clearly display the various components of the target furniture and label them accordingly.

[0084] In this embodiment, various practical applications can be performed based on the component segmentation diagram and component labels of the target furniture. In an exemplary application scheme, after completing the annotation of a large number of furniture images, a furniture screening request can be received, and the screening request can include a target component label, such as a European-style armrest, etc.; in this way, the target furniture image with the target component label can be selected from the large number of furniture images and output as the screening result. In another exemplary application scheme, after completing the annotation of a large number of furniture images, a furniture design request for the target furniture image can be received, and the furniture design request includes a component label that needs to be modified, such as an armrest; in this way, according to the component segmentation diagram corresponding to the target furniture image and the design action required in the furniture design request, the armrest in the target furniture image can be cut, replaced / moved, and other design operations can be performed. Of course, these application schemes are exemplary, and this embodiment is not limited thereto.

[0085] Based on this, in this embodiment, starting from a 3D furniture model, based on point cloud data, the component labels belonging to each spatial point in the 3D furniture model can be determined. Based on this, the 3D furniture model can be converted into a 2D furniture image, and the component labels already labeled in the 3D furniture model can be mapped to the corresponding pixels in the 2D furniture image. The 2D furniture image can then be recolored based on coloring parameters configured for different component labels to generate a component segmentation map corresponding to the 3D furniture model. In this way, the component labels determined in the 3D furniture model can be presented in the 2D furniture image in the form of color. Thus, for the same 3D furniture model, a 2D furniture image, a component segmentation map, and component labels can be obtained. These can be used as training samples to input into a furniture component segmentation model, allowing the furniture component segmentation model to learn how to segment components in 2D furniture images. Consequently, the trained furniture component segmentation model can be used to annotate component labels on any 2D furniture image, thereby enabling automatic component-level labeling of massive furniture images, saving significant manpower and resources and overcoming the labeling challenges in furniture scenes.

[0086] It should be noted that the execution entity of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution entity of steps 100 to 104 can be device A; for another example, the execution entity of steps 101 and 102 can be device A, and the execution entity of step 103 can be device B; and so on.

[0087] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 801 and 802, are merely used to distinguish between different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0088] Figure 5 A schematic diagram of a computing device provided as another exemplary embodiment of the present application is shown in FIG. Figure 5 As shown, the computing device includes a memory 50 and a processor 51 .

[0089] The processor 51 is coupled to the memory 50 and is configured to execute the computer program in the memory 50 to:

[0090] Obtain point cloud data corresponding to the 3D furniture model;

[0091] Based on the point cloud data, determine the component labels of each spatial point contained in the 3D furniture model;

[0092] Convert 3D furniture models into 2D furniture images;

[0093] In the 2D furniture image, based on the mapping relationship between pixels and spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels, the 2D furniture image is recolored to generate a component segmentation map corresponding to the 3D furniture model;

[0094] The 2D furniture images, component segmentation maps and component labels corresponding to the 3D furniture models are used as training samples to train the furniture component segmentation model.

[0095] In an optional embodiment, when the processor 51 determines the component label to which each spatial point included in the 3D furniture model belongs, it may be configured to:

[0096] Input point cloud data into the point cloud segmentation model;

[0097] In the point cloud segmentation model, clustering is performed on each spatial point contained in the 3D furniture model to obtain at least one spatial point set;

[0098] Performing attribute feature extraction on at least one spatial point set according to the point cloud data to obtain attribute features corresponding to the at least one spatial point set;

[0099] Based on the mapping relationship between the attribute features and the component labels and the attribute features corresponding to the at least one spatial point set, the component label to which the at least one spatial point set belongs is output.

[0100] In an optional embodiment, during the process of training the point cloud segmentation model, the processor 51 may be configured to:

[0101] Obtain several labeled 3D furniture samples, which contain point cloud data and component labels for each spatial point in the 3D model.

[0102] Input several marked 3D furniture samples into the point cloud segmentation model;

[0103] In the point cloud segmentation model, clustering and attribute feature extraction are performed on spatial points in the 3D model;

[0104] Based on the extracted attribute features and the component labels to which each spatial point belongs, the mapping relationship between the attribute features and the component labels is learned.

[0105] In an optional embodiment, when clustering the spatial points included in the 3D furniture model, the processor 51 may be configured to:

[0106] Clustering is performed on each spatial point included in the 3D furniture model according to the coordinates and / or texture data of the spatial point.

[0107] In an optional embodiment, when converting the 3D furniture model into a 2D furniture image according to the point cloud data, the processor 51 is configured to:

[0108] Rasterizing the 3D furniture model according to the point cloud data to map the spatial points in the world coordinate system contained in the 3D furniture model into pixel points in the view coordinate system to obtain a two-dimensional image;

[0109] According to the texture data of the triangular mesh where the spatial points contained in the 3D furniture model are located, the corresponding pixel points in the two-dimensional image are rendered to generate a 2D furniture image corresponding to the 3D furniture model.

[0110] In an optional embodiment, when recoloring the 2D furniture image based on the mapping relationship between the pixel points and the spatial points, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels, the processor 51 is configured to:

[0111] Determine a target spatial point associated with a first pixel point in the 2D furniture image based on a mapping relationship between pixel points and spatial points;

[0112] Determine the target component label corresponding to the first pixel point according to the component label to which the target spatial point belongs;

[0113] Calculating a pixel value of a first pixel according to a coloring parameter configured for the target component label;

[0114] Recoloring the first pixel according to the pixel value of the first pixel;

[0115] The first pixel point is any pixel point in the 2D furniture image.

[0116] In an alternative embodiment, different component labels are configured with different coloring parameters.

[0117] In an optional embodiment, the component label includes one or more of a component name label, a component texture label, a component shape label, or a component material label.

[0118] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can be referred to the relevant descriptions in the above-mentioned embodiments of the training scheme for the furniture parts segmentation model. In order to save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this application.

[0119] based on Figure 5 The structure of the computing device shown in FIG. 1 may further provide another computing device in another exemplary embodiment of the present application.

[0120] In this embodiment, the processor 51 in the computing device may be configured to:

[0121] Obtain a two-dimensional furniture image corresponding to the target furniture;

[0122] Input the 2D furniture image into the furniture component segmentation model. The furniture component segmentation model is trained using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples. The component segmentation map is obtained by recoloring the 2D furniture image based on the mapping relationship between pixels in the 2D furniture image and spatial points in the 3D furniture model, the component labels to which the spatial points belong, and the coloring parameters configured for different component labels.

[0123] In the furniture parts segmentation model, the two-dimensional furniture image is segmented to generate a parts segmentation map corresponding to the two-dimensional furniture image;

[0124] Each segmented area in the component segmentation diagram is marked with a component label.

[0125] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can be referred to the relevant descriptions in the above-mentioned embodiments of the furniture image marking scheme. In order to save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this application.

[0126] Further, if Figure 5 As shown, the computing device also includes: a communication component 52, a power supply component 53 and other components. Figure 5 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 5 Components shown.

[0127] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be executed by a computing device in the above method embodiment.

[0128] above Figure 5 The memory in the computing platform is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0129] above Figure 5 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0130] above Figure 5 The power supply component in a device provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0131] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0132] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0135] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0136] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0137] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0139] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A method for training a furniture parts segmentation model, comprising: Obtain point cloud data corresponding to the 3D furniture model; Based on the point cloud data, determining the component labels of the spatial points included in the 3D furniture model; Converting the 3D furniture model into a 2D furniture image; Under the 2D furniture image, based on a mapping relationship between pixel points and spatial points, determining a target spatial point associated with a first pixel point under the 2D furniture image, where the first pixel point is any pixel point in the 2D furniture image; Determine the component label to which the target spatial point belongs as the target component label corresponding to the first pixel point; Calculating a pixel value of the first pixel point according to a coloring parameter configured for the target component label; Recoloring the first pixel point according to the pixel value of the first pixel point to generate a component segmentation map corresponding to the 3D furniture model; The 2D furniture images, component segmentation maps, and component labels corresponding to the 3D furniture models are used as training samples to train a furniture component segmentation model.

2. The method according to claim 1, wherein determining the component label of each spatial point included in the 3D furniture model based on the point cloud data comprises: Inputting the point cloud data into a point cloud segmentation model; In the point cloud segmentation model, clustering the spatial points included in the 3D furniture model to obtain at least one spatial point set; performing attribute feature extraction on the at least one spatial point set according to the point cloud data to obtain attribute features corresponding to the at least one spatial point set; Based on the mapping relationship between the attribute features and the component labels and the attribute features corresponding to the at least one spatial point set, the component label to which the at least one spatial point set belongs is output.

3. The method according to claim 2, wherein the training process of the point cloud segmentation model comprises: Obtaining a number of labeled 3D furniture samples, wherein the 3D furniture samples include point cloud data and component labels belonging to each spatial point in the 3D model; Inputting the plurality of marked 3D furniture samples into the point cloud segmentation model; In the point cloud segmentation model, clustering and attribute feature extraction are performed on the spatial points in the 3D model; Based on the extracted attribute features and the component labels to which each spatial point belongs, a mapping relationship between the attribute features and the component labels is learned.

4. The method according to claim 2, wherein clustering the spatial points contained in the 3D furniture model comprises: Clustering is performed on each spatial point included in the 3D furniture model according to the coordinates and / or texture data of the spatial point.

5. The method according to claim 1, wherein converting the 3D furniture model into a 2D furniture image comprises: rasterizing the 3D furniture model according to the point cloud data to map spatial points in a world coordinate system contained in the 3D furniture model into pixel points in a view coordinate system to obtain a two-dimensional image; According to the texture data of the triangular mesh where the spatial point included in the 3D furniture model is located, the corresponding pixel points in the two-dimensional image are rendered to generate a 2D furniture image corresponding to the 3D furniture model. The method according to claim 1 , wherein different component labels are configured with different coloring parameters. 7 . The method according to claim 1 , wherein the component label comprises one or more of a component name label, a component texture label, a component shape label, or a component material label.

8. A method for marking a furniture image, comprising: Obtain a two-dimensional furniture image corresponding to the target furniture; Inputting the two-dimensional furniture image into a furniture component segmentation model, wherein the furniture component segmentation model is obtained by training using the 2D furniture image, component segmentation map, and component labels corresponding to the 3D furniture model as training samples, and the component segmentation map is based on a mapping relationship between pixel points in the 2D furniture image and spatial points of the 3D furniture model, and determines a target spatial point associated with a first pixel point in the 2D furniture image; Determine the component label to which the target spatial point belongs as the target component label corresponding to the first pixel point; Calculating a pixel value of the first pixel point according to a coloring parameter configured for the target component label; obtained by recoloring the first pixel point according to the pixel value of the first pixel point; In the furniture component segmentation model, image segmentation is performed on the two-dimensional furniture image to generate a component segmentation map corresponding to the two-dimensional furniture image; Wherein, each segmented area in the component segmentation diagram is marked with a component label.

9. A computing device comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is configured to execute the one or more computer instructions to execute the method for training a furniture component segmentation model according to any one of claims 1 to 7.

10. A computing device comprising a memory and a processor; The memory is used to store one or more computer instructions; The processor is coupled to the memory and is configured to execute the one or more computer instructions to execute the furniture image marking method according to claim 8.

Images