Automatic labeling method and device for three-dimensional model, equipment and storage medium

Abstract

The invention provides an automatic labeling method and device for a three-dimensional model, equipment and a storage medium. Relates to the technical field of computers, in particular to the technical fields of 3D model marking, data processing, model marking and the like. According to the specific implementation scheme, under the visual angles of a plurality of virtual cameras, mask graphs of semantic tags in a to-be-labeled composite model are obtained, and the to-be-labeled composite model is a three-dimensional model comprising a plurality of semantics; determining respective component channel diagrams of the sampling point set of the to-be-labeled composite model under the visual angles of the plurality of virtual cameras; and based on the component channel graphs and the mask graphs under the visual angles of the plurality of virtual cameras, marking semantic tags to which each component in the to-be-marked composite model belongs.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to technical fields such as 3D (3 Dimensions) model annotation, data processing, model marking, etc. Background Art

[0002] A 3D model is a digital representation of an object in 3D space, describing the shape, structure, material properties, etc. of the object through a series of geometric elements (such as points, lines, surfaces, volumes, etc.) and their mutual relationships. It has characteristics such as intuitiveness and rich information, and plays an important role in many industries.

[0003] For example, in the customized home furnishing industry, users can design a 3D model of home furnishings based on an existing standardized home furnishing model library according to their needs for home furnishing functions and appearance. The designed 3D model can be rendered for discussing the design plan. Moreover, it can also be reused in other home decoration design plans. Summary of the Invention

[0004] The present disclosure provides an automatic annotation method, device, equipment and storage medium for 3D models to solve or alleviate one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an automatic annotation method for 3D models, including: Under multiple virtual camera perspectives, respectively obtain mask maps of each semantic label in the composite model to be annotated, where the composite model to be annotated is a 3D model including multiple semantics; Determine the respective component channel maps of the sampling point set of the composite model to be annotated under multiple virtual camera perspectives; Based on the component channel maps and mask maps under multiple virtual camera perspectives, annotate the semantic labels to which each component in the composite model to be annotated belongs.

[0006] In a second aspect, the present disclosure provides an automatic annotation device for 3D models, including: An acquisition module, configured to respectively obtain mask maps of each semantic label in the composite model to be annotated under multiple virtual camera perspectives, where the composite model to be annotated is a 3D model including multiple semantics; A determination module, configured to determine the respective component channel maps of the sampling point set of the composite model to be annotated under multiple virtual camera perspectives; An annotation module, configured to annotate the semantic labels to which each component in the composite model to be annotated belongs based on the component channel maps and mask maps under multiple virtual camera perspectives.

[0007] In a third aspect, there is provided an electronic device, including: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute any method in the embodiments of the present disclosure.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method in the embodiments of the present disclosure.

[0009] In a fifth aspect, a computer program product is provided, including a computer program which, when executed by a processor, implements any method in the embodiments of the present disclosure.

[0010] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings, unless otherwise specified, the same reference numerals throughout the several views denote the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0012] Figure 1 is a schematic flow chart of an automatic annotation method for a three-dimensional model according to the first embodiment of the present disclosure; Figure 2 is a schematic flow chart of determining multiple virtual camera viewpoints according to the second embodiment of the present disclosure; Figure 3 is a schematic flow chart of obtaining a mask graph of each semantic label in a composite model to be annotated according to the third embodiment of the present disclosure; Figure 4 is a schematic flow chart of generating a component channel graph for each virtual camera viewpoint according to the fourth embodiment of the present disclosure; Figure 5 is a schematic flow chart of annotating the labels to which each component in the composite model to be annotated belongs according to the fifth embodiment of the present disclosure; Figure 6 is an example diagram of a voting matrix according to the sixth embodiment of the present disclosure; Figure 7 is a schematic flow chart of generating an independent model and associating a corresponding semantic label with the independent model according to the seventh embodiment of the present disclosure; Figure 8It is a schematic structural diagram of an automatic annotation device for a 3D model according to the eighth embodiment of the present disclosure; Figure 9 It is a block diagram of an electronic device for implementing the automatic annotation method of the 3D model according to the embodiment of the present disclosure. Detailed implementation manners

[0013] Hereinafter, the present disclosure will be further described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings do not have to be drawn to scale unless otherwise specified.

[0014] In addition, for better illustration of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can be implemented without some specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0015] Furthermore, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present disclosure, "a plurality of" means two or more unless otherwise specifically defined.

[0016] It should be noted that for the operations shown in the flowcharts in the embodiments of the present disclosure, unless it is explicitly stated that there is a sequence of execution between different operations, or different operations have a sequence of execution in technical implementation, the sequence of execution between multiple operations may be unordered, and multiple operations may also be executed simultaneously.

[0017] In the field of 3D model related applications, such as 3D model generation, scene editing (especially scenes that require fine operation of the model like home decoration design, etc.), there are key requirements for the accuracy and independence of 3D models.

[0018] However, there are currently a large number of composite models in the model library that are composed of multiple independent models. When users perform scene editing based on independent models, such as replacing specific components, they cannot accurately locate and perform the replacement operation. This problem directly leads to a reduction in the convenience of user operations, and at the same time limits the flexibility of model editing, making it difficult to meet the customized requirements of diverse scenes.

[0019] In the currently common model processing methods, the composite model can be labeled manually to distinguish different independent models in the composite model. However, this method has many drawbacks. On the one hand, when different personnel perform labeling, due to inconsistent understandings of semantics and grasps of standards, it is difficult to unify the labeling standards, resulting in uneven quality of model data. On the other hand, the manual labeling process is time-consuming and laborious, with low efficiency. In the face of a large-scale model library, it is difficult to meet the requirements of rapid processing and application.

[0020] In view of this, an automatic labeling method for three-dimensional models is proposed in the embodiments of the present disclosure. This method is applicable to three-dimensional models in any field, such as three-dimensional models of toy products, home furnishings, and equipment.

[0021] As Figure 1 shown, it is a schematic flowchart of this method, including the following content: S101, under multiple virtual camera viewpoints, respectively obtain the mask images of each semantic label in the composite model to be labeled, where the composite model to be labeled is a three-dimensional model including multiple semantics.

[0022] A semantic label refers to a description of the function, category, or use of each component in the composite model to be labeled. For example, in a three-dimensional model of a composite piece of furniture to be labeled consisting of a table and a chair, both the table and the chair correspond to their respective semantic labels.

[0023] A virtual camera viewpoint refers to the angle and position of simulating a real camera in a virtual environment. By setting multiple different virtual camera viewpoints, the mask images of each semantic label in the composite model to be labeled can be obtained respectively. This mask image is two-dimensional image information generated by virtual cameras from different viewpoints and can be presented by a binary image (black and white image). For example, the white area indicates the model area corresponding to the semantic label in this viewpoint, and the black area is the background or other non-target areas.

[0024] S102, determine the respective part channel images of the sampling point set of the composite model to be labeled under multiple virtual camera viewpoints.

[0025] Since the labeling only focuses on which semantic label each component of the composite model belongs to, the sampling point set can be a series of points selected from the surface of the three-dimensional model to represent the key positions of different components of the entire composite model. By processing and analyzing these sampling points, it helps to determine the semantic labels to which each component of the composite model to be labeled belongs.

[0026] A part channel image is an image generated for each sampling point in the sampling point set under a virtual camera viewpoint, indicating the position and distribution of the part to which each sampling point belongs in this viewpoint, and is used to subsequently combine with the mask image to determine the semantic label corresponding to each component.

[0027] S103. Label the semantic tags to which each component in the composite model to be labeled belongs based on the component channel maps and mask maps under multiple virtual camera views.

[0028] By combining the component channel maps and mask maps obtained from multiple virtual camera views, the semantic tags to which each component in the entire composite model to be labeled belongs are inferred.

[0029] In the embodiments of the present disclosure, under multiple virtual camera views, mask maps of each semantic tag in the composite model to be labeled are respectively obtained, which can observe the model from different angles, comprehensively capture the information of each component in the composite model to be labeled, reduce the problem of information loss of some components caused by occlusion, and thus provide a richer and more complete data basis for subsequent accurate labeling. The mask map can clearly separate the components with specific semantic tags from the background and other components. The component channel map can represent the components to which the sampling point set of the composite model to be labeled belongs under multiple virtual camera views. Through the mask map and the component channel map, the association relationships among the view, semantics, sampling points, and components can be established. Therefore, labeling the semantic tags to which each component in the composite model to be labeled belongs based on the component channel maps and mask maps under multiple virtual camera views can comprehensively capture the feature information of the 3D model. The information under different views complements each other, reduces problems such as occlusion and information loss that may exist in a single view, and thus more accurately determines the boundaries and features of each component, improving the accuracy and efficiency of automatic labeling.

[0030] In the embodiments of the present disclosure, to improve the accuracy of automatic labeling, mask maps of each semantic tag in the composite model to be labeled can be respectively obtained under multiple virtual camera views.

[0031] During implementation, the multiple virtual camera views can be determined first based on the following method, as Figure 2 shown, including: S201. Taking the target anchor point of the composite model to be labeled as a reference, establish multiple view ranges where the virtual camera views are aligned with the target anchor point.

[0032] The target anchor point is a selected key point in the composite model to be labeled. During implementation, the composite model can be abstracted as a sphere, with the center point of the sphere as the target anchor point, and this anchor point is used as the focusing reference for the virtual camera views. Of course, points within a specified area from this center point can also be selected as the target anchor point, as long as the appearance state of the composite model to be labeled can be completely observed based on the target anchor point under multiple virtual camera views. The embodiments of the present disclosure do not limit the determination method of the target anchor point.

[0033] Establish multiple viewing ranges for the virtual camera to align with the target anchor point, that is, covering the target anchor point from multiple angles in a multi-angle manner to comprehensively observe the area around the target anchor point from different directions. Exemplarily, the virtual camera can be set based on the horizontal plane where the center of the sphere is located and at least one preset position (such as positions r / 2 above and r / 2 below) with respect to this horizontal plane.

[0034] S202, sample within the multiple viewing ranges to obtain multiple virtual camera viewpoints.

[0035] That is, from the multiple viewing ranges where the virtual camera viewpoints are aligned with the target anchor point, a set of positions and directions of the virtual cameras are selected by sampling to obtain diverse viewing angles. These selected viewpoints will be directly applied to the subsequent model rendering and semantic annotation processes.

[0036] For example, the viewpoints are evenly distributed at an angular interval of a preset angle (such as 15 degrees), and each viewpoint of the virtual camera is aligned with the center point of the sphere.

[0037] It can be understood that sampling is an operation to select specific virtual camera poses within the viewing range. During implementation, it can be completed through random sampling, uniform sampling, or other sampling algorithms.

[0038] In the embodiments of the present disclosure, establishing the viewing range based on the target anchor point can ensure that the virtual camera viewpoints focus on the key areas of the model. Establishing multiple viewing ranges and sampling within them can observe the composite model to be annotated from different angles and directions. It can ensure that all parts of the composite model to be annotated can be observed, avoiding omissions or occlusion problems caused by a single viewpoint, so as to comprehensively obtain the information of the composite model to be annotated, provide a richer data basis for subsequent accurate annotation, and improve the accuracy of automatic annotation.

[0039] During implementation, for each virtual camera viewpoint, obtain the mask map of each semantic label in the composite model to be annotated, as Figure 3 shown, including: S301, render the composite model to be annotated under the virtual camera viewpoint to obtain the to-be-processed rendering map corresponding to the virtual camera viewpoint.

[0040] Rendering refers to the process of converting a 3D model into a 2D image. Under the virtual camera viewpoint, rendering the composite model to be annotated can generate a model image seen from a specific virtual camera viewpoint through factors such as the virtual camera position and viewpoint, and use this model image as the to-be-processed rendering map corresponding to the virtual camera viewpoint.

[0041] S302, input the to-be-processed rendering map into the target detection model to obtain the target detection results corresponding to each semantic label in the composite model to be annotated.

[0042] A target detection model is a computer vision model used to identify and locate instances of specific objects or semantic categories in an input image. In the embodiments of the present disclosure, the target detection model may be GroundingDINO (Marrying DINO with Grounded Pre - Training for Open - Set Object Detection, a cross - modal target detection model based on DINO).

[0043] Input the rendering graph to be processed into the target detection model. The target detection model will analyze the rendering graph to be processed and output the bbox (Bounding Box) containing the target, the corresponding semantic label category, confidence, etc.

[0044] During implementation, for the rendering graph to be processed under the perspective of the virtual camera, input the rendering graph to be processed into the target detection model to obtain the target detection results corresponding to each semantic label in the composite model to be annotated. This can be achieved based on the following steps: Step A1: Input the rendering graph to be processed into the target detection model to obtain multiple initial detection boxes corresponding to each semantic label respectively. The initial detection box represents the position range of each semantic label initially detected by the target model in the rendering graph to be processed. Since there may be different position ranges corresponding to the same semantic label in the rendering graph to be processed, multiple initial detection boxes of the same semantic label overlap. To improve the automatic annotation efficiency, the most accurate detection result can be determined through further processing.

[0045] Step A2: Based on the non - maximum suppression method, screen out the target detection boxes corresponding to each semantic label from the multiple initial detection boxes to obtain the target detection results corresponding to each semantic label respectively.

[0046] Non - Maximum Suppression (NMS) is a commonly used post - processing technique in computer vision, mainly used to solve the problem of overlapping of multiple detection boxes in the target detection task.

[0047] When a target detection model generates initial detection boxes, it assigns a confidence score to each detection box, indicating the likelihood that the detection box contains the components corresponding to the semantic labels. All the initial detection boxes with the same semantic label are sorted in descending order of confidence. The detection box with the highest confidence is selected as a retained target detection box. Then, the Intersection over Union (IOU) is used to calculate the overlap degree between other detection boxes and this retained detection box. If the IOU of a certain detection box and the retained target detection box exceeds the set threshold, it is considered that these two detection boxes are detecting the same component, and this detection box is discarded; otherwise, this detection box is retained. Repeat this process until all the initial detection boxes are processed to obtain accurate detection results for each semantic label.

[0048] S303, input the target detection result into the segmentation model to obtain the mask maps corresponding to each semantic label respectively.

[0049] A segmentation model is a model that can assign each pixel in an image to a specific semantic label. It usually outputs a segmentation mask, where the position of each pixel is associated with a semantic label.

[0050] In the embodiments of the present disclosure, the segmentation model can be a SAM-2 (Segment Anything Model 2, a segmentation model) model.

[0051] The segmentation model will further perform precise pixel-level segmentation on the target object (such as an independent model corresponding to a semantic label) according to the target detection result to determine which semantic label each pixel belongs to. The finally output mask map can clearly show the specific regions corresponding to each semantic label in the image, providing an accurate data basis for subsequent annotation.

[0052] In the embodiments of the present disclosure, for each virtual camera view, the composite model to be annotated is rendered separately. The rendered images from different views can provide different feature information of each component. Input the rendered image to be processed into the target detection model. The target detection model can accurately locate the bounding boxes of these labels by learning the appearance features of the semantic labels, thereby clarifying the positional relationship of one or more components corresponding to each semantic label in the composite model. Therefore, through the accurate mask map associated with the semantic label, it is possible to provide a data reference for automatic annotation and improve the accuracy of automatic annotation.

[0053] To further complete the annotation of the composite model to be annotated, component channel maps corresponding to the sampling point set of the model can be generated under multiple different virtual camera views to reflect the distribution of each component of the model under different views.

[0054] During implementation, determine the component channel maps of the sampling point set of the composite model to be labeled under multiple virtual camera views. The sampling point set can be obtained by sampling the composite model to be labeled; the sampling point set is used to describe the overall appearance characteristics of the composite model to be labeled in the world coordinate system.

[0055] The sampling point set is a set of representative points selected from the composite model to be labeled. The set formed by combining these points is used to describe the overall appearance characteristics of the composite model to be labeled in the world coordinate system. The world coordinate system is a fixed global coordinate system, and the composite model to be labeled has a definite position and orientation in this coordinate system.

[0056] During implementation, for each virtual camera view among multiple virtual camera views, obtaining the sampling point set can be achieved based on the following steps: Step B1, under the virtual camera view, collect the depth map of the composite model to be labeled; The depth map is a special image that records the depth information of each pixel point in the three-dimensional model observed from the virtual camera view to the camera plane. It is usually presented in the form of a two-dimensional image, and each pixel value in the image represents the depth of the corresponding position of the pixel.

[0057] Since each virtual camera view has its own local coordinate system, in order to unify the depth map information under different views, it is necessary to convert the depth map collected under each virtual camera view to the world coordinate system.

[0058] During implementation, in step B2, the depth map can be converted to the world coordinate system to summarize the depth maps of the composite model to be labeled in the world coordinate system under multiple virtual camera views, and obtain a depth map set; That is, each point in the depth map is converted from the local coordinate system of the virtual camera to the world coordinate system. The depth values under multiple virtual camera views after coordinate conversion are summarized together to form a depth map set.

[0059] Step B3, sample the depth map set to obtain the sampling point set.

[0060] That is, through the sampling operation, a certain number of points are selected from the depth map set, and these points constitute the sampling point set.

[0061] After obtaining the sampling point set, for each virtual camera view, the implementation method of generating the component channel map for each virtual camera view can be as Figure 4 shown, including: S401, map the sampling point set to the rendering graph to be processed of the composite model to be labeled under the virtual camera view, and obtain the mapped point set of the sampling point set in the rendering graph to be processed.

[0062] During implementation, the sampling points are transformed into the camera coordinate system through the extrinsic and intrinsic parameters of the virtual camera. At the same time, the image center point of the composite model to be labeled in the camera coordinate system is aligned with the center point of the coordinate system of the composite model to be labeled, so as to obtain the mapped points of the sampling points under the to-be-processed rendering image. After this mapping, the pixel positions corresponding to these mapped points can be determined in the to-be-processed rendering image, and these pixel positions constitute the set of mapped points in the to-be-processed rendering image.

[0063] S402. Based on the component information of each mapped point in the set of mapped points in the composite model to be labeled, determine the component labels of each sampling point in the set of sampling points.

[0064] The composite model to be labeled itself contains the information of each component. After the sampling points are mapped into the to-be-processed rendering image, according to the position of the mapped points in the to-be-processed rendering image and combined with the component information of the model, it can be judged which component the mapped point belongs to, and then the corresponding component label is assigned to the corresponding sampling point.

[0065] S403. Based on the component labels of each sampling point, generate a component channel map under the virtual camera view.

[0066] When generating the component channel map, each sampling point in the set of sampling points is marked as the corresponding part according to the component label of each sampling point. During implementation, different components can be represented by different colors or gray values, so as to visually observe the distribution of each component of the composite model to be labeled in the set of sampling points in the component channel map.

[0067] In the embodiments of the present disclosure, a set of sampling points is obtained by sampling the composite model to be labeled, and these sampling points can describe the overall appearance characteristics of the composite model to be labeled in the world coordinate system. Through the sampling method, the main geometric shape characteristics of the composite model to be labeled and the positional relationship between different sampling points can be summarized with fewer sampling points, realizing the dimensionality reduction representation of the model, which is convenient for subsequent processing and analysis. Mapping the set of sampling points into the to-be-processed rendering images under different virtual camera views establishes the connection between the sampling points in the world coordinate system and the images under each virtual camera view. Different virtual camera views can provide multi-faceted information of the model. Through this mapping, the overall characteristics of the composite model to be labeled in the world coordinate system can be associated with the image characteristics of each virtual machine view, providing richer information for subsequent analysis and processing. Generating a component channel map corresponding to the virtual camera view based on the component labels of each sampling point, and the component channel map can visually display the distribution and characteristics of each component of the model's sampling points under different views. Through the component channel map, it is possible to quickly identify the components to which the sampling points in the composite model to be labeled belong, so as to improve the accuracy of automatic labeling.

[0068] In the embodiments of the present disclosure, in the component channel map, each component corresponds to an image channel respectively, and different components are expressed using different channel values.

[0069] In image processing, an image channel is an independent data layer for storing information about a certain aspect of an image. Common image types include single-channel and three-channel. Each channel can be regarded as a two-dimensional array, and each element in the array represents a specific component attribute value at the corresponding position in the image.

[0070] In the component channel map, each component is assigned to an independent image channel. That is, different sampling points in the sampling point set of the composite model to be labeled will be mapped to the corresponding component channels for separate representation under the corresponding virtual camera viewpoints.

[0071] The channel value refers to the numerical value of each pixel point in the channel. In the embodiments of the present disclosure, it is used to represent the component to which the sampling point or the mapped point under the corresponding virtual camera viewpoint belongs on the corresponding channel. In the component channel map, different components are distinguished by different channel values. Exemplarily, for the channel corresponding to a certain component, the channel value of the pixel position belonging to the component is set to a fixed non-zero value (such as 255), and the channel value of the pixel position not belonging to the component is set to 0.

[0072] In the embodiments of the present disclosure, assigning different components to different image channels and giving different channel values can make the information of each component clearly presented in the channel map. Different components are expressed using different channel values to facilitate the semantic labeling of the combined mask map and automatically label each component.

[0073] Based on the foregoing, after obtaining the component channel maps and mask maps under multiple virtual camera viewpoints, the labels to which each component in the composite model to be labeled belongs can be labeled based on the component channel maps and mask maps under multiple virtual camera viewpoints. The specific implementation manner is as Figure 5 shown and includes: S'501, based on the mask maps corresponding to the component channel maps of each component under multiple virtual camera viewpoints respectively, vote on the semantic labels to which each component belongs to obtain the voting probabilities of each component belonging to each semantic label.

[0074] Under multiple virtual camera viewpoints, each component of the composite model to be labeled corresponds to a component channel map and a mask map. Among them, the component channel map describes the components to which the sampling points that can be observed when observing the sampling point set under the virtual camera viewpoint, that is, when observing the composite model to be labeled, are projected into the two-dimensional space, and the mask map is used to indicate the semantic labels of each part of the composite model to be labeled projected into the two-dimensional space under the same virtual camera viewpoint. Thus, an association relationship between the component and the semantic label is established in the same spatial expression.

[0075] During implementation, obtain the voting probabilities of each component belonging to each semantic label. For example, Figure 5 as shown, it can be specifically implemented as follows: S5011, Initialize the voting matrix. The first direction in the voting matrix includes multiple semantic labels in the composite model to be labeled. Each semantic label respectively corresponds to its own second direction. Each component respectively corresponds to an element in the second direction, and each element is initialized to a default value.

[0076] For example, Figure 6 as shown, initialize the voting matrix. This matrix has two directions. The first direction includes multiple semantic labels in the composite model to be labeled. For example, "table", "chair", etc. are each a semantic label. The second direction includes each component in the composite model to be labeled. For example, "table leg 1", "table leg 2", "table leg 3", "table leg 4", etc. Each component corresponds to an element in the second direction, and each element is initialized to a default value, such as 0.5.

[0077] Of course, Figure 6 only some components are shown in the figure. When a composite model to be labeled includes a complete table and a complete chair, the components also include other parts. For example, in addition to the chair legs, the chair further includes components such as the backrest and the seat cushion.

[0078] Therefore, it can be understood that during implementation, which specific semantic labels and components are included are determined by the composite model to be labeled.

[0079] S5012, Perform iterative operations on multiple virtual camera views to obtain the voting probabilities of each component belonging to each semantic label.

[0080] During implementation, the iterative operations to be performed may include the following steps: Step C1, For each component in the current voting matrix, determine the mask graph of the virtual camera view to which the component channel graph of the component in the virtual camera view belongs, and obtain the target mask graph of the component; Based on the foregoing, for each component, there is a corresponding component channel graph in the virtual camera view. By finding the mask graph corresponding to the component in that view according to the virtual camera view where the component channel graph is located, and taking it as the target mask graph. For example, both chair leg 1 and chair leg 2 of the chair may correspond to the same mask graph, that is, the mask graph of the chair. Therefore, the component channel graphs of each component may be a part of the corresponding mask graph.

[0081] It can be understood that the part channel map is obtained by sampling the composite model to be labeled, mapping the sampled points to the mapped points under the same virtual camera view, and dividing them according to the parts to which they belong. Under the same virtual camera view, each point can also be divided based on the mask map. Thus, the association relationship between the semantic labels represented by the part channel map and the mask map is established.

[0082] Step C2: Optimize the value of the target element in the current voting matrix according to the first strategy so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is the element corresponding to the semantic label of the target mask map to which the part belongs. That is, when the part belongs to the semantic label corresponding to the target mask map, the value of the target element is optimized according to the first strategy. The first strategy can be adding a first step value to the value of the target element, or multiplying by a coefficient greater than 1, etc., so that the optimized value of the target element is higher than the value before optimization, that is, higher than the value of the target element in the current voting matrix.

[0083] Step C3: Optimize the other elements in the current voting matrix except the target element in the first direction according to the second strategy so that the values of the other elements are lower than the values of the other elements in the current voting matrix.

[0084] Correspondingly, in the first direction of the voting matrix, the other elements except the target element are the "other elements". The values of these other elements can be optimized according to the second strategy so that the optimized values of these other elements are lower than the values before optimization, that is, lower than the values of these other elements in the current voting matrix. Common second strategies can be multiplying the value of the other element by a coefficient less than 1 or subtracting a second step value, etc.

[0085] Among them, the first step value and the second step value can be the same or different, and the first step value and the second step value can be fixed values or determined dynamically according to the actual situation.

[0086] Continuously repeat the above iterative operation to obtain the voting probabilities of each part belonging to each semantic label.

[0087] In the embodiments of the present disclosure, the voting probability reflects the likelihood of each part belonging to each semantic label. By comprehensively considering the voting results from multiple virtual camera views, the obtained voting probability can more comprehensively and accurately determine the semantic label of the part, thereby improving the efficiency and accuracy of automatic annotation.

[0088] S502: For each part, based on the maximum value of the voting probabilities of the part belonging to each semantic label, determine the semantic label to which the part belongs.

[0089] In the embodiments of the present disclosure, by performing semantic label voting on the mask maps of each component from multiple virtual camera perspectives, information from different perspectives can be comprehensively considered, providing a more sufficient basis for determining semantic labels. There may be differences in the judgment of component semantics from different perspectives. Through the voting mechanism, the judgments from multiple perspectives can be mutually verified, avoiding incorrect judgments caused by similar semantic label probabilities, and improving the accuracy of determining the semantic label corresponding to the component as the label to which the component belongs.

[0090] In the embodiments of the present disclosure, after determining the semantic label corresponding to the component, it can be combined into an independent model to facilitate scene editing based on this independent model. The implementation method can be as Figure 7 shown, including: S701, Extract components with the same semantic label from the composite model to be labeled.

[0091] Traverse all components in the composite model to be labeled and classify them according to the semantic label to which each component belongs. For example, if the semantic labels are "table" and "chair", all components belonging to "table" are extracted and placed in one group, and components belonging to "chair" are placed in another group.

[0092] S702, Combine components with the same semantic label into an independent model and associate the corresponding semantic label with the independent model.

[0093] Combine components such as "table leg 1", "table leg 2", "table leg 3", "table leg 4", and "tabletop" with the semantic label of "table" into an independent model, that is, an independent table, and associate the semantic label of "table".

[0094] In the embodiments of the present disclosure, in the original complex model, components with the same semantic label may be distributed in different positions and need to be labeled one by one. After extracting and combining them, the corresponding semantic label can be associated with the independent model, and then the labeling result can be applied to the corresponding components in the original model, avoiding repeated labor and improving the labeling efficiency.

[0095] In the embodiments of the present disclosure, each semantic label can independently correspond to an independent model. For example, when the composite model to be labeled includes a table and four chairs, the semantic labels can be marked as chair 1, chair 2, chair 3, and chair 4. Therefore, based on the method provided by the embodiments of the present disclosure, different independent models (semantic labels) can be accurately identified and automatic labeling can be completed.

[0096] Based on the same technical concept, the embodiments of the present disclosure also provide an automatic labeling device 800 for a three-dimensional model, as Figure 8 shown, including: An acquisition module 801, configured to acquire mask graphs of each semantic label in a to-be-annotated composite model respectively under multiple virtual camera perspectives, where the to-be-annotated composite model is a three-dimensional model including multiple semantics; A determination module 802, configured to determine respective part channel graphs of a sampling point set of the to-be-annotated composite model under multiple virtual camera perspectives; An annotation module 803, configured to annotate semantic labels to which each part in the to-be-annotated composite model belongs based on the part channel graphs and mask graphs under multiple virtual camera perspectives.

[0097] In some embodiments, the acquisition module is specifically configured to: For each virtual camera perspective, perform the following operations respectively: Render the to-be-annotated composite model under the virtual camera perspective to obtain a to-be-processed rendered image corresponding to the virtual camera perspective; Input the to-be-processed rendered image into an object detection model to obtain object detection results corresponding to each semantic label in the to-be-annotated composite model; Input the object detection results into a segmentation model to obtain mask graphs corresponding to each semantic label respectively.

[0098] In some embodiments, it further includes a perspective determination module, configured to determine multiple virtual camera perspectives based on the following method: Taking the target anchor point of the to-be-annotated composite model as a reference, establish multiple perspective ranges where the virtual camera perspectives are aligned with the target anchor point; Sample within the multiple perspective ranges to obtain multiple virtual camera perspectives.

[0099] In some embodiments, the determination module is specifically configured to: Sample the to-be-annotated composite model to obtain a sampling point set; the sampling point set is used to describe the overall appearance characteristics of the to-be-annotated composite model in the world coordinate system; For each virtual camera perspective, perform the following operations respectively: Map the sampling point set into the to-be-processed rendered image of the to-be-annotated composite model under the virtual camera perspective to obtain a set of mapped points of the sampling point set in the to-be-processed rendered image; Based on the part information of each mapped point in the mapped point set in the to-be-annotated composite model, determine the part labels of each sampling point in the sampling point set; Generate a part channel graph under the virtual camera perspective based on the part labels of each sampling point.

[0100] In some embodiments, in the part channel graph, each part corresponds to an image channel respectively, and different parts are expressed with different channel values.

[0101] In some embodiments, the annotation module includes: A voting unit configured to vote on the semantic labels to which each component belongs based on the mask graphs corresponding to the component channel graphs of each component from multiple virtual camera perspectives, to obtain the voting probabilities of each component belonging to each semantic label; A determination unit configured to determine, for each component, the semantic label to which the component belongs based on the maximum value of the voting probabilities of the component belonging to each semantic label.

[0102] In some embodiments, the voting unit is specifically configured to: Initialize a voting matrix, where the first direction in the voting matrix includes multiple semantic labels in the composite model to be annotated, each semantic label respectively corresponds to its own second direction, each component respectively corresponds to an element in the second direction, and each element is initialized to a default value; Iteratively perform the following operations on multiple virtual camera perspectives to obtain the voting probabilities of each component belonging to each semantic label: For each component in the current voting matrix, determine the mask graph in the virtual camera perspective to which the component channel graph of the component in the virtual camera perspective belongs, to obtain the target mask graph of the component; Optimize the value of the target element in the current voting matrix according to a first strategy, so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is the element corresponding to the semantic label to which the target mask graph to which the component belongs; Optimize the other elements in the first direction in the current voting matrix except the target element according to a second strategy, so that the values of the other elements are lower than the values of the other elements in the current voting matrix.

[0103] In some embodiments, it further includes a combination module configured to extract components with the same semantic label from the composite model to be annotated; Combine the components with the same semantic label into an independent model, and associate the corresponding semantic label with the independent model.

[0104] For the specific functions and examples of each module and sub-module of the device according to the embodiments of the present disclosure, reference may be made to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be elaborated herein.

[0105] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0106] Figure 9 It is a structural block diagram of an electronic device according to an embodiment of the present disclosure. As Figure 9As shown in the figure, the electronic device includes: a memory 910 and a processor 920. The memory 910 stores a computer program that can run on the processor 920. The number of the memory 910 and the processor 920 can be one or more. The memory 910 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device executes the method provided in the above method embodiment. The electronic device may further include: a communication interface 930, configured to communicate with external devices and perform data interaction and transmission.

[0107] If the memory 910, the processor 920, and the communication interface 930 are implemented independently, the memory 910, the processor 920, and the communication interface 930 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 9 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0108] Optionally, in a specific implementation, if the memory 910, the processor 920, and the communication interface 930 are integrated on a chip, the memory 910, the processor 920, and the communication interface 930 can communicate with each other through an internal interface.

[0109] It should be understood that the above-mentioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It is worth noting that the processor can be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0110] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may further include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0111] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present disclosure are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (e.g., infrared, Bluetooth, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., Digital Versatile Disc (DVD)), or a semiconductor medium (e.g., Solid State Disk (SSD)), etc. It should be noted that the computer-readable storage medium mentioned in the present disclosure can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0112] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk, an optical disc, or the like.

[0113] In the description of the embodiments of the present disclosure, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0114] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means "or". For example, A / B may mean A or B. "And / or" herein is merely a description of the relationship between associated objects, indicating that there can be three relationships. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0115] In the description of the embodiments of the present disclosure, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, "a plurality of" means two or more.

[0116] The above are only exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. An automatic annotation method for 3D models, comprising: Obtaining mask graphs of each semantic label in a composite model to be annotated from multiple virtual camera perspectives, where the composite model to be annotated is a 3D model including multiple semantics; Determining component channel graphs of a sampling point set of the composite model to be annotated from the multiple virtual camera perspectives respectively; Based on the component channel graphs and mask graphs from the multiple virtual camera perspectives, annotating semantic labels to which each component in the composite model to be annotated belongs.

2. The method according to claim 1, wherein, The step of obtaining mask graphs of each semantic label in a composite model to be annotated from multiple virtual camera perspectives includes: For each virtual camera perspective, respectively performing the following operations: Rendering the composite model to be annotated from the virtual camera perspective to obtain a to-be-processed rendering graph corresponding to the virtual camera perspective; Inputting the to-be-processed rendering graph into an object detection model to obtain object detection results corresponding to each semantic label in the composite model to be annotated; Inputting the object detection results into a segmentation model to obtain mask graphs corresponding to each semantic label respectively.

3. The method according to claim 1 or 2, further comprising determining the multiple virtual camera perspectives based on the following method: Taking a target anchor point of the composite model to be annotated as a reference, establishing multiple perspective ranges for virtual camera perspectives to align with the target anchor point; Sampling within the multiple perspective ranges to obtain the multiple virtual camera perspectives.

4. The method according to claim 1, wherein The step of determining component channel graphs of a sampling point set of the composite model to be annotated from the multiple virtual camera perspectives respectively includes: Sampling the composite model to be annotated to obtain the sampling point set; the sampling point set is used to describe the overall appearance features of the composite model to be annotated in the world coordinate system; For each virtual camera perspective respectively performing the following operations: Mapping the sampling point set into a to-be-processed rendering graph of the composite model to be annotated from the virtual camera perspective to obtain a set of mapped points of the sampling point set in the to-be-processed rendering graph; Based on component information of each mapped point in the mapped point set in the composite model to be annotated, determining component labels of each sampling point in the sampling point set; Generating a component channel graph from the virtual camera perspective based on the component labels of each sampling point.

5. The method according to claim 1 or 4, wherein, In the component channel graph, each component corresponds to an image channel respectively, and different components are expressed by different channel values.

6. The method according to claim 1, wherein, The step of annotating semantic labels to which each component in the composite model to be annotated belongs based on the component channel graphs and mask graphs from the multiple virtual camera perspectives includes: Voting on semantic labels to which each component belongs based on mask graphs respectively corresponding to the component channel graphs of each component from the multiple virtual camera perspectives to obtain voting probabilities of each component belonging to each semantic label; For each component, determining the semantic label to which the component belongs based on the maximum value of the voting probabilities of the component belonging to each semantic label.

7. The method according to claim 6, wherein, Voting on the semantic labels to which each component belongs based on the mask graphs corresponding to the component channel graphs of each component under the multiple virtual camera views, to obtain the voting probabilities of each component belonging to each semantic label, including: Initializing a voting matrix, where the first direction in the voting matrix includes multiple semantic labels in the composite model to be labeled, each semantic label corresponds to its own second direction respectively, each component corresponds to an element in the second direction, and each element is initialized to a default value; Iteratively performing the following operations on the multiple virtual camera views to obtain the voting probabilities of each component belonging to each semantic label: For each component in the current voting matrix, determining the mask graph under the virtual camera view to which the component channel graph of the component under the virtual camera view belongs, to obtain the target mask graph of the component; Optimizing the value of the target element in the current voting matrix according to a first strategy, so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is the element corresponding to the semantic label of the target mask graph to which the component belongs; Optimizing the other elements in the current voting matrix except the target element in the first direction according to a second strategy, so that the values of the other elements are lower than the values of the other elements in the current voting matrix.

8. The method according to claim 1, further comprising: Extracting components with the same semantic label from the composite model to be labeled; Combining the components with the same semantic label into an independent model, and associating the corresponding semantic label with the independent model.

9. An automatic annotation device for a three-dimensional model, comprising: An acquisition module, configured to respectively acquire mask graphs of each semantic label in a composite model to be labeled under multiple virtual camera views, where the composite model to be labeled is a three-dimensional model including multiple semantics; A determination module, configured to determine the component channel graphs of the sampling point set of the composite model to be labeled under the multiple virtual camera views respectively; A labeling module, configured to label the semantic labels to which each component in the composite model to be labeled belongs based on the component channel graphs and mask graphs under the multiple virtual camera views.

10. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the method according to any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

12. A computer program product, comprising a computer program, where the computer program implements the method according to any one of claims 1-8 when executed by a processor.

Images