Automatic annotation method, device, equipment and storage medium for three-dimensional models

Abstract

The present disclosure provides a method, apparatus, device and storage medium for automatic annotation of three-dimensional models. The present disclosure relates to the field of computer technology, and in particular to technical fields such as 3D model annotation, data processing, and model labeling. The specific implementation scheme is as follows: under multiple virtual camera perspectives, respectively obtain mask maps of each semantic label in the composite model to be annotated, where the composite model to be annotated is a three-dimensional model including multiple semantics; determine the component channel maps of the sampling point set of the composite model to be annotated under multiple virtual camera perspectives; based on the component channel maps and mask maps under multiple virtual camera perspectives, annotate the semantic labels to which each component in the composite model to be annotated belongs.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as 3D (3 Dimensions) model annotation, data processing, and model marking. Background Art

[0002] A 3D model is a digital representation of an object in three-dimensional space. It describes the object's shape, structure, material, and other properties through a series of geometric elements (such as points, lines, surfaces, and volumes) and their interrelationships. Its intuitive and information-rich nature makes it a crucial tool in numerous industries.

[0003] For example, in the custom home furnishing industry, users can design a 3D model of a home based on their functional and aesthetic requirements, drawing from an existing library of standardized home models. The resulting 3D model can be rendered for design discussion and reused in other home furnishing design projects. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, device and storage medium for automatic annotation of a three-dimensional model to solve or alleviate one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a method for automatically labeling a three-dimensional model, comprising:

[0006] Obtaining, from multiple virtual camera perspectives, mask images of various semantic labels in the composite model to be labeled, where the composite model to be labeled is a three-dimensional model including multiple semantics;

[0007] Determine the component channel graphs of the sampling point set of the composite model to be annotated under multiple virtual camera perspectives;

[0008] Based on the component channel maps and mask maps under multiple virtual camera perspectives, the semantic labels belonging to each component in the composite model to be annotated are annotated.

[0009] In a second aspect, the present disclosure provides an automatic annotation device for a three-dimensional model, comprising:

[0010] An acquisition module is used to respectively acquire, from multiple virtual camera perspectives, a mask image of each semantic label in a composite model to be annotated, where the composite model to be annotated is a three-dimensional model including multiple semantics;

[0011] A determination module, used to determine the component channel graphs of the sampling point set of the composite model to be annotated under multiple virtual camera perspectives;

[0012] The annotation module is used to annotate the semantic labels of each component in the composite model to be annotated based on the component channel map and mask map under multiple virtual camera perspectives.

[0013] According to a third aspect, an electronic device is provided, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0017] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0018] In a fifth aspect, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments provided in accordance with the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0021] Figure 1 is a flowchart of a method for automatically labeling a three-dimensional model according to the first embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of a process for determining multiple virtual camera perspectives according to the second embodiment of the present disclosure;

[0023] Figure 3 is a schematic diagram of a process for obtaining a mask map of each semantic label in a composite model to be annotated according to the third embodiment of the present disclosure;

[0024] Figure 4 is a schematic diagram of a process for generating a component channel diagram under each virtual camera perspective according to a fourth embodiment of the present disclosure;

[0025] Figure 5is a schematic diagram of a process for labeling labels belonging to various components in a composite model to be labeled according to the fifth embodiment of the present disclosure;

[0026] Figure 6 is an example diagram of a voting matrix according to the sixth embodiment of the present disclosure;

[0027] Figure 7 is a flowchart of generating an independent model and associating corresponding semantic tags for the independent model according to the seventh embodiment of the present disclosure;

[0028] Figure 8 is a schematic structural diagram of an automatic annotation device for a three-dimensional model according to an eighth embodiment of the present disclosure;

[0029] Figure 9 It is a block diagram of an electronic device used to implement the automatic annotation method of a three-dimensional model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The present disclosure will be described in further detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0031] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, circuits, etc. well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0033] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present disclosure, or there is a sequence of execution between different operations in technical implementation, otherwise, the execution order between multiple operations may not be prioritized, and multiple operations may also be executed simultaneously.

[0034] In 3D model-related application fields, such as 3D model generation and scene editing (especially scenes such as home decoration design that require precise model manipulation), there are key requirements for the accuracy and independence of 3D models.

[0035] However, the current model library contains a large number of composite models composed of multiple independent models. When users edit scenes based on independent models, such as replacing specific components, they cannot accurately locate and execute the replacement operation. This problem directly reduces user convenience and limits the flexibility of model editing, making it difficult to meet the customization needs of diverse scenarios.

[0036] Among the common current model processing methods, manual labeling is used to distinguish the individual models within a composite model. However, this approach has numerous drawbacks. Firstly, due to inconsistent understanding of semantics and adherence to standards among different labelers, it is difficult to standardize labeling standards, resulting in inconsistent model data quality. Secondly, manual labeling is time-consuming, labor-intensive, and inefficient, making it difficult to meet the requirements of rapid processing and application for large-scale model libraries.

[0037] In view of this, the present disclosure provides an automatic annotation method for 3D models. This method is applicable to 3D models in any field, such as 3D models of toys, home furnishings, and equipment.

[0038] like Figure 1 As shown, it is a flow chart of the method, which includes the following contents:

[0039] S101 , obtaining mask images of various semantic labels in a composite model to be labeled under multiple virtual camera perspectives, where the composite model to be labeled is a three-dimensional model including multiple semantics.

[0040] Semantic labels describe the function, category, or purpose of each component in the composite model to be annotated. For example, in a 3D composite furniture model consisting of a table and a chair, each component would have its own semantic label.

[0041] A virtual camera perspective simulates the angle and position of a real camera in a virtual environment. By setting multiple virtual camera perspectives, masks can be obtained for each semantic label in the composite model to be annotated. This mask is a two-dimensional image generated by virtual cameras at different perspectives and can be represented as a binary image (black and white). For example, white areas indicate the model area corresponding to the semantic label at that perspective, while black areas represent background or other non-target areas.

[0042] S102 , determining component channel graphs of respective sampling point sets of the composite model to be annotated under multiple virtual camera perspectives.

[0043] Since annotation only focuses on which semantic label each component of the composite model belongs to, the sampling point set can be a series of points selected from the surface of the three-dimensional model to represent the key positions of different components of the entire composite model. By processing and analyzing these sampling points, it helps to determine the semantic labels to which each component of the composite model to be annotated belongs.

[0044] The component channel map is an image generated for each sampling point in the sampling point set under the perspective of a virtual camera. It represents the position and distribution of the component to which each sampling point belongs under this perspective, and is subsequently used in conjunction with the mask map to determine the semantic label corresponding to each component.

[0045] S103 , based on the component channel graphs and mask graphs under the perspectives of multiple virtual cameras, annotate the semantic labels to which each component in the composite model to be annotated belongs.

[0046] By combining the component channel maps and mask maps obtained from multiple virtual camera perspectives, the semantic labels of each component in the entire composite model to be annotated can be inferred.

[0047] In the disclosed embodiments, mask images for each semantic label in the composite model to be annotated are obtained from multiple virtual camera perspectives. This allows the model to be observed from different angles, comprehensively capturing information about each component of the composite model to be annotated and reducing the problem of missing information about certain components due to occlusion, thereby providing a richer and more complete data foundation for subsequent accurate annotation. The mask image can clearly separate components with specific semantic labels from the background and other components. The component channel map can represent the components to which the sampling points of the composite model to be annotated belong under multiple virtual camera perspectives. Through the mask image and component channel map, the relationship between perspective, semantics, sampling points, and components can be established. Therefore, based on the component channel map and mask image from multiple virtual camera perspectives, the semantic labels belonging to each component in the composite model to be annotated are annotated, which can comprehensively capture the feature information of the three-dimensional model. The information from different perspectives complements each other, reducing the problems of occlusion and missing information that may exist in a single perspective, thereby more accurately determining the boundaries and features of each component, improving the accuracy and efficiency of automatic annotation.

[0048] In the embodiment of the present disclosure, in order to improve the accuracy of automatic annotation, mask images of various semantic labels in the composite model to be annotated may be obtained respectively under multiple virtual camera perspectives.

[0049] During implementation, multiple virtual camera perspectives can be determined based on the following methods: Figure 2 As shown, including:

[0050] S201 , using the target anchor point of the composite model to be annotated as a reference, establishing multiple viewing angle ranges of a virtual camera that is aligned with the target anchor point.

[0051] The target anchor point is a key point selected within the composite model to be annotated. During implementation, the composite model can be abstracted as a sphere, with the center point of the sphere serving as the target anchor point, and this anchor point serving as the focus reference for the virtual camera's perspective. Alternatively, a point within a specified area from the center point can be selected as the target anchor point. As long as the appearance of the composite model to be annotated can be fully observed from multiple virtual camera perspectives using the target anchor point as a reference, the disclosed embodiments do not limit the method for determining the target anchor point.

[0052] Establish multiple viewing angles of the virtual camera focused on the target anchor point, i.e., providing multiple angles with the target anchor point as the center, to comprehensively observe the area surrounding the target anchor point from different directions. For example, the virtual camera can be set based on the horizontal plane at the center of the sphere and at least one preset position relative to that horizontal plane (e.g., positions r / 2 above and r / 2 below).

[0053] S202: Sampling is performed within multiple viewing angles to obtain multiple virtual camera viewing angles.

[0054] This involves sampling a set of virtual camera positions and orientations from multiple viewpoints, creating a diverse range of viewing angles. These selected viewpoints are then directly applied to the subsequent model rendering and semantic annotation processes.

[0055] For example, the viewing angles are evenly distributed at intervals of a preset angle (such as 15 degrees), and each viewing angle of the virtual camera is aligned with the center point of the sphere.

[0056] It is understood that sampling is the operation of selecting a specific virtual camera pose from the viewing angle range. During implementation, it can be accomplished through random sampling, uniform sampling, or other sampling algorithms.

[0057] In the disclosed embodiment, establishing a viewing angle range based on the target anchor point ensures that the virtual camera's perspective focuses on key areas of the model. By establishing multiple viewing angle ranges and sampling within them, the composite model to be annotated can be observed from different angles and directions. This ensures that all parts of the composite model to be annotated are visible, avoiding omissions or occlusions caused by a single viewing angle. This allows for comprehensive information about the composite model to be annotated, providing a richer data foundation for subsequent accurate annotation and improving the accuracy of automatic annotation.

[0058] During implementation, for each virtual camera perspective, obtain the mask map of each semantic label in the composite model to be annotated, such as Figure 3 As shown, including:

[0059] S301 , rendering the composite model to be annotated under the perspective of a virtual camera to obtain a rendering image to be processed corresponding to the perspective of the virtual camera.

[0060] Rendering is the process of converting a 3D model into a 2D image. Rendering the composite model to be annotated from a virtual camera's perspective generates an image of the model as seen from a specific virtual camera perspective, taking into account factors such as the virtual camera's position and viewing angle. This image serves as the rendering to be processed corresponding to the virtual camera's perspective.

[0061] S302: Input the rendered image to be processed into the target detection model to obtain target detection results corresponding to the semantic labels in the composite model to be labeled.

[0062] An object detection model is a computer vision model used to identify and locate specific objects or instances of semantic categories in an input image. In the disclosed embodiments, the object detection model may be GroundingDINO (Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, a cross-modal object detection model based on DINO).

[0063] The rendered image to be processed is input into the target detection model. The target detection model will analyze the rendered image to be processed and output the bounding box (Bounding Box) containing the target, the corresponding semantic label category, the confidence level, etc.

[0064] During implementation, for the unprocessed rendering image under the perspective of the virtual camera, the unprocessed rendering image is input into the target detection model to obtain the target detection results corresponding to each semantic label in the composite model to be annotated. This can be achieved based on the following steps:

[0065] Step A1: Input the rendered image to be processed into the object detection model to obtain multiple initial detection boxes corresponding to each semantic label;

[0066] The initial detection box represents the location range of each semantic label initially detected by the target model within the rendered image. Because the same semantic label may appear in different location ranges within the rendered image, multiple initial detection boxes for the same semantic label may overlap. To improve the efficiency of automatic labeling, further processing is performed to determine the most accurate detection result.

[0067] In step A2, based on the non-maximum suppression method, target detection frames corresponding to the respective semantic labels are screened out from the multiple initial detection frames to obtain target detection results corresponding to the respective semantic labels.

[0068] Non-Maximum Suppression (NMS) is a post-processing technique commonly used in computer vision, mainly used to solve the problem of overlapping multiple detection boxes in target detection tasks.

[0069] When generating initial detection frames, the object detection model assigns each frame a confidence score, indicating the likelihood that the frame contains the component corresponding to the semantic label. All initial detection frames with the same semantic label are sorted from high to low by confidence. The frame with the highest confidence score is selected as the retained object detection frame. The intersection over union (IOU) is then used to calculate the overlap between the remaining detection frames and the retained detection frame. If the IOU of a detection frame with the retained object detection frame exceeds a set threshold, the two frames are considered to be detecting the same component and are discarded. Otherwise, the frame is retained. This process is repeated until all initial detection frames have been processed, resulting in accurate detection results for each semantic label.

[0070] S303: Input the target detection result into the segmentation model to obtain the mask map corresponding to each semantic label.

[0071] A segmentation model is a model that can assign each pixel in an image to a specific semantic label. It usually outputs a segmentation mask where each pixel position is associated with a semantic label.

[0072] In the embodiment of the present disclosure, the segmentation model may be a SAM-2 (Segment Anything Model 2, a segmentation model) model.

[0073] Based on the object detection results, the segmentation model further performs precise pixel-level segmentation of the target object (e.g., a separate model corresponding to a semantic label), determining which semantic label each pixel belongs to. The resulting mask clearly displays the specific regions within the image corresponding to each semantic label, providing an accurate data foundation for subsequent annotation.

[0074] In the disclosed embodiment, the composite model to be labeled is rendered separately for each virtual camera perspective. Renderings from different perspectives can provide different feature information for each component. The rendered image to be processed is input into the target detection model. The target detection model can accurately locate the bounding boxes of these labels by learning the appearance features of the semantic labels, thereby clarifying the positional relationship of one or more components corresponding to each semantic label in the composite model. Therefore, through accurate mask maps associated with semantic labels, a data reference for automatic labeling can be provided, thereby improving the accuracy of automatic labeling.

[0075] In order to further complete the annotation of the composite model to be annotated, corresponding component channel maps can be generated for the sampling point set of the model under multiple different virtual camera perspectives to reflect the distribution of each component of the model under different perspectives.

[0076] During implementation, the component channel diagrams of the sampling point set of the composite model to be labeled under multiple virtual camera perspectives are determined. The sampling point set can be obtained by sampling the composite model to be labeled; the sampling point set is used to describe the overall appearance characteristics of the composite model to be labeled in the world coordinate system.

[0077] The sampling point set is a collection of representative points selected from the composite model to be annotated. These points are combined to form a set that describes the overall appearance of the composite model in the world coordinate system. The world coordinate system is a fixed global coordinate system in which the composite model to be annotated has a specific position and posture.

[0078] During implementation, for each virtual camera perspective in the plurality of virtual camera perspectives, obtaining a set of sampling points may be achieved based on the following steps:

[0079] Step B1: Acquire a depth map of the composite model to be annotated from the perspective of a virtual camera;

[0080] A depth map is a special image that records the depth information from each pixel in a three-dimensional model observed from the perspective of a virtual camera to the camera plane. It is usually presented in the form of a two-dimensional image, and each pixel value in the image represents the depth of the corresponding position of the pixel.

[0081] Since each virtual camera perspective has its own local coordinate system, in order to unify the depth map information under different perspectives, the depth map collected under each virtual camera perspective needs to be converted to the world coordinate system.

[0082] During implementation, in step B2, the depth map may be converted into a world coordinate system to aggregate the depth maps of the composite model to be annotated in the world coordinate system under the perspectives of multiple virtual cameras to obtain a depth map set;

[0083] That is, each point in the depth map is converted from the local coordinate system of the virtual camera to the world coordinate system. The depth values ​​from multiple virtual camera perspectives after coordinate conversion are aggregated to form a depth map set.

[0084] Step B3: Sampling the depth map set to obtain a sampling point set.

[0085] That is, through the sampling operation, a certain number of points are selected from the depth map set, and these points constitute the sampling point set.

[0086] After obtaining the sampling point set, for each virtual camera perspective, the implementation method of generating the component channel map under each virtual camera perspective can be as follows: Figure 4 As shown, including:

[0087] S401 : Mapping a sampling point set to a to-be-processed rendering image of a composite model to be annotated under a virtual camera perspective to obtain a mapping point set of the sampling point set in the to-be-processed rendering image.

[0088] During implementation, the sampling points are converted to the camera coordinate system using the virtual camera's extrinsic and intrinsic parameters. At the same time, the center point of the composite model to be annotated in the camera coordinate system is aligned with the center point of the composite model's coordinate system. This results in the mapping points of the sampling points in the rendered image to be processed. After this mapping, the pixel locations corresponding to these mapping points can be determined in the rendered image to be processed. These pixel locations constitute the set of mapping points in the rendered image to be processed.

[0089] S402 : Determine a component label for each sampling point in the sampling point set based on component information of each mapping point in the mapping point set in the composite model to be labeled.

[0090] The composite model to be labeled contains the information of each component. When the sampling point is mapped to the rendering to be processed, the position of the mapping point in the rendering to be processed and the component information of the model can be used to determine which component the mapping point belongs to, and then assign the corresponding component label to the corresponding sampling point.

[0091] S403: Generate a component channel map from the perspective of a virtual camera based on the component labels of each sampling point.

[0092] When generating a component channel map, each sampling point in the sample set is labeled as a corresponding part based on its component label. Different colors or grayscale values ​​can be used to represent different parts, allowing the component channel map to visually visualize the distribution of the components of the composite model to be labeled within the sample point set.

[0093] In the disclosed embodiments, a composite model to be annotated is sampled to obtain a set of sampling points. These sampling points can describe the overall appearance characteristics of the composite model to be annotated in a world coordinate system. Through sampling, the main geometric features of the composite model to be annotated, as well as the positional relationships between different sampling points, can be summarized with a relatively small number of sampling points, achieving a dimensionality-reduced representation of the model and facilitating subsequent processing and analysis. The set of sampling points is mapped onto the rendered images to be processed from different virtual camera perspectives, establishing a connection between the sampling points in the world coordinate system and the images from each virtual camera perspective. Different virtual camera perspectives can provide multiple aspects of the model. Through this mapping, the overall characteristics of the composite model to be annotated in the world coordinate system can be associated with the image characteristics from each virtual camera perspective, providing richer information for subsequent analysis and processing. Based on the component labels of each sampling point, a component channel map for the corresponding virtual camera perspective is generated. The component channel map can intuitively display the distribution and characteristics of each component of the model's sampling points from different perspectives. The component channel map can quickly identify the component to which each sampling point in the composite model to be annotated belongs, thereby improving the accuracy of automatic annotation.

[0094] In the embodiment of the present disclosure, in the component channel graph, each component corresponds to an image channel, and different components are expressed using different channel values.

[0095] In image processing, an image channel is an independent data layer used to store information about a specific aspect of an image. Common image types include single-channel and three-channel. Each channel can be thought of as a two-dimensional array, where each element represents the attribute value of a specific component at the corresponding position in the image.

[0096] In the component channel graph, each component is assigned to a separate image channel. That is, different sampling points in the sample point set of the composite model to be annotated are mapped to the corresponding component channel for separate representation under the corresponding virtual camera perspective.

[0097] A channel value refers to the numerical value of each pixel in a channel. In the disclosed embodiments, it is used to represent the component to which the sampling point or the mapping point under the corresponding virtual camera perspective belongs on the corresponding channel. In the component channel diagram, different components are distinguished by different channel values. For example, for a channel corresponding to a component, the channel value of the pixel position belonging to the component is set to a fixed non-zero value (e.g., 255), and the channel value of the pixel position not belonging to the component is set to 0.

[0098] In this disclosed embodiment, different components are assigned to different image channels and assigned different channel values, allowing the information of each component to be clearly presented in the channel map. Using different channel values ​​to represent different components facilitates automatic labeling of each component using the semantic labels of the combined mask map.

[0099] Based on the above, after obtaining the component channel map and mask map under multiple virtual camera perspectives, the labels to which each component in the composite model to be labeled can be marked based on the component channel map and mask map under multiple virtual camera perspectives. The specific implementation method is as follows Figure 5 As shown, including:

[0100] S501 : Based on the mask images corresponding to the component channel images of each component under multiple virtual camera perspectives, vote on the semantic labels to which each component belongs to, and obtain the voting probability that each component belongs to each semantic label.

[0101] Under multiple virtual camera perspectives, each component of the composite model to be annotated is mapped to a component channel map and a mask map. The component channel map describes the component to which each observed sampling point, projected into two-dimensional space, belongs when observing a set of sampling points from the virtual camera perspective—that is, when observing the composite model to be annotated. The mask map indicates the semantic labels of each part of the composite model to be annotated projected into this two-dimensional space from the same virtual camera perspective. This establishes an association between components and semantic labels within the same spatial representation.

[0102] During implementation, the voting probability of each component belonging to each semantic label is obtained, such as Figure 5 As shown, it can be implemented as follows:

[0103] S5011, initialize the voting matrix. The first direction in the voting matrix includes multiple semantic labels in the composite model to be annotated. Each semantic label corresponds to its own second direction. Each component corresponds to an element in the second direction. Each element is initialized to a default value.

[0104] like Figure 6 As shown, a voting matrix is ​​initialized with two directions. The first direction contains multiple semantic labels in the composite model to be annotated, for example, "table," "chair," and so on, each with a semantic label. The second direction contains the components in the composite model to be annotated, for example, "table leg 1," "table leg 2," "table leg 3," "table leg 4," and so on. Each component corresponds to an element in the second direction, and each element is initialized to a default value, for example, 0.5.

[0105] certainly, Figure 6 Only some of the components are shown. When a composite model to be annotated includes a complete table and a complete chair, the components also include other parts. For example, a chair includes not only the legs, but also the backrest, seat cushion, and other parts.

[0106] Therefore, it is understandable that, during implementation, the specific semantic labels and components included are determined by the composite model to be annotated.

[0107] S5012: Perform an iterative operation on multiple virtual camera perspectives to obtain the voting probability that each component belongs to each semantic label.

[0108] When implementing, the iterative operations to be performed may include the following steps:

[0109] Step C1: for each component in the current voting matrix, determine the mask map under the virtual camera perspective to which the component channel map under the virtual camera perspective belongs, and obtain the target mask map of the component;

[0110] As mentioned above, each component has a corresponding component channel map under the virtual camera's viewpoint. Based on the virtual camera's viewpoint at which the component channel map resides, the mask map corresponding to the component at that viewpoint is found and used as the target mask map. For example, legs 1 and 2 of a chair may both correspond to the same mask map: the chair's mask map. Therefore, the component channel map of each component may be part of the corresponding mask map.

[0111] The component channel map can be understood as a collection of sampled composite models to be annotated. The sampled points are then mapped to the mapped points from the same virtual camera perspective, and the resulting map is divided according to the components they belong to. Within the same virtual camera perspective, points can also be divided based on a mask map. This establishes an association between the semantic labels represented by the component channel map and the mask map.

[0112] Step C2: Optimizing the value of the target element in the current voting matrix according to the first strategy so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is the element of the semantic label corresponding to the target mask image to which the component belongs;

[0113] That is, when the component belongs to the semantic label corresponding to the target mask image, the value of the target element is optimized according to the first strategy. The first strategy can be to add the first step length value to the value of the target element, or multiply it by a coefficient greater than 1, so that the optimized value of the target element is higher than the value before optimization, that is, higher than the value of the target element in the current voting matrix.

[0114] Step C3: Optimize the other elements in the current voting matrix except the target element in the first direction according to the second strategy, so that the values ​​of the other elements are lower than the values ​​of the other elements in the current voting matrix.

[0115] Accordingly, in the first direction of the voting matrix, the elements other than the target element are referred to as "other elements." The values ​​of these other elements can be optimized according to the second strategy, such that the optimized values ​​of these other elements are lower than their pre-optimization values, that is, lower than the values ​​of these other elements in the current voting matrix. Common second strategies include multiplying the values ​​of the other elements by a coefficient less than 1 or subtracting a second step size.

[0116] The first step length value and the second step length value may be the same or different. The first step length value and the second step length value may be fixed values ​​or may be dynamically determined according to actual conditions.

[0117] The above iterative operation is repeated continuously to obtain the voting probability of each component belonging to each semantic label.

[0118] In this disclosed embodiment, the voting probability reflects the likelihood that each component belongs to each semantic label. By integrating the voting results from multiple virtual camera perspectives, the resulting voting probability can more comprehensively and accurately determine the semantic label of the component, thereby improving the efficiency and accuracy of automatic labeling.

[0119] S502 : For each component, determine the semantic label to which the component belongs based on the maximum value of the voting probabilities that the component belongs to each semantic label.

[0120] In the disclosed embodiments, semantic label voting is performed on the mask images of each component from multiple virtual camera perspectives. This allows for comprehensive consideration of information from different perspectives, providing a more comprehensive basis for determining semantic labels. The judgment of component semantics may differ from perspective to perspective. The voting mechanism allows these judgments to be mutually verified, avoiding misjudgments caused by similar semantic label probabilities and improving the accuracy of determining the semantic label corresponding to a component as the label to which the component is assigned.

[0121] In the embodiment of the present disclosure, after determining the semantic labels corresponding to the components, they can be combined into an independent model to facilitate scene editing based on the independent model. Figure 7 As shown, including:

[0122] S701: extract components with the same semantic labels from the composite model to be annotated.

[0123] Traverse all components in the composite model to be annotated and classify them according to their semantic labels. For example, if the semantic labels are "table" and "chair," extract all components that are classified as "table" and place them in one group, and components that are classified as "chair" and place them in another group.

[0124] S702: Combining components with the same semantic label into an independent model, and associating the independent model with a corresponding semantic label.

[0125] An independent model consisting of components such as "table leg 1", "table leg 2", "table leg 3", "table leg 4", and "table top" with the semantic label "table" is created, i.e., an independent table, and is associated with the semantic label "table".

[0126] In the disclosed embodiments, components with the same semantic label may be located in different locations within the original complex model and require individual annotation. After extracting and combining these components, the corresponding semantic labels can be associated with the independent models. The annotation results can then be applied to the corresponding components in the original model, avoiding duplication of effort and improving annotation efficiency.

[0127] In the disclosed embodiments, each semantic label can independently correspond to an independent model. For example, if the composite model to be annotated includes a table and four chairs, the semantic labels can be chair 1, chair 2, chair 3, and chair 4. Therefore, the method provided by the disclosed embodiments can accurately identify the different independent models (semantic labels) and complete automatic labeling.

[0128] Based on the same technical concept, the embodiment of the present disclosure also provides an automatic annotation device 800 for a three-dimensional model. Figure 8 As shown, including:

[0129] An acquisition module 801 is configured to acquire, from multiple virtual camera perspectives, mask images of various semantic labels in a composite model to be labeled, where the composite model to be labeled is a three-dimensional model including multiple semantics.

[0130] A determination module 802 is configured to determine component channel graphs of respective sampling point sets of the composite model to be annotated under multiple virtual camera perspectives;

[0131] The labeling module 803 is used to label the semantic labels of each component in the composite model to be labeled based on the component channel map and mask map under multiple virtual camera perspectives.

[0132] In some embodiments, the acquisition module is specifically configured to:

[0133] For each virtual camera perspective, perform the following operations:

[0134] Render the composite model to be annotated under the perspective of the virtual camera to obtain a rendering to be processed corresponding to the perspective of the virtual camera;

[0135] Input the rendered image to be processed into the target detection model to obtain the target detection results corresponding to each semantic label in the composite model to be labeled;

[0136] The target detection results are input into the segmentation model to obtain the mask maps corresponding to each semantic label.

[0137] In some embodiments, the system further includes a perspective determination module for determining multiple virtual camera perspectives based on the following method:

[0138] Taking the target anchor point of the composite model to be annotated as a reference, establish multiple viewing angle ranges of the virtual camera's viewing angle aligned with the target anchor point;

[0139] Sampling is performed within multiple viewing angles to obtain multiple virtual camera viewing angles.

[0140] In some embodiments, the determining module is specifically configured to:

[0141] Sampling the composite model to be labeled to obtain a set of sampling points; the set of sampling points is used to describe the overall appearance characteristics of the composite model to be labeled in the world coordinate system;

[0142] For each virtual camera viewpoint, perform the following operations:

[0143] Mapping the sampling point set to a to-be-processed rendering of the composite model to be annotated under the perspective of a virtual camera to obtain a mapping point set of the sampling point set in the to-be-processed rendering;

[0144] Determine a component label for each sampling point in the sampling point set based on component information of each mapping point in the mapping point set in the composite model to be labeled;

[0145] Based on the component labels of each sampling point, a component channel map from the perspective of the virtual camera is generated.

[0146] In some embodiments, in the component channel graph, each component corresponds to an image channel, and different components are expressed using different channel values.

[0147] In some embodiments, the annotation module includes:

[0148] The voting unit is used to vote for the semantic label to which each component belongs based on the mask images corresponding to the component channel images of each component under multiple virtual camera perspectives, and obtain the voting probability of each component belonging to each semantic label;

[0149] The determining unit is configured to determine, for each component, the semantic label to which the component belongs based on the maximum value of the voting probabilities that the component belongs to each semantic label.

[0150] In some embodiments, the voting unit is specifically configured to:

[0151] Initialize a voting matrix, where the first direction of the voting matrix includes multiple semantic labels in the composite model to be annotated, each semantic label corresponds to its own second direction, each component corresponds to an element in the second direction, and each element is initialized to a default value;

[0152] Perform the following operations iteratively for multiple virtual camera perspectives to obtain the voting probability of each component belonging to each semantic label:

[0153] For each component in the current voting matrix, determine the mask map under the virtual camera perspective to which the component channel map under the virtual camera perspective belongs, and obtain the target mask map of the component;

[0154] Optimizing the value of the target element in the current voting matrix according to the first strategy so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is the element of the semantic label corresponding to the target mask image to which the component belongs;

[0155] Other elements in the current voting matrix except the target element in the first direction are optimized according to the second strategy, so that the values ​​of other elements are lower than the values ​​of other elements in the current voting matrix.

[0156] In some embodiments, a combination module is further included for extracting components with the same semantic label from the composite model to be annotated;

[0157] Components with the same semantic labels are combined into independent models, and the corresponding semantic labels are associated with the independent models.

[0158] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0159] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0160] Figure 9 FIG. 1 is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 9 As shown, the electronic device includes: a memory 910 and a processor 920. The memory 910 stores a computer program that can be executed on the processor 920. The number of memory 910 and processor 920 can be one or more. The memory 910 can store one or more computer programs. When the one or more computer programs are executed by the electronic device, the electronic device performs the method provided by the above method embodiment. The electronic device may also include: a communication interface 930 for communicating with external devices and performing data exchange.

[0161] If the memory 910, processor 920, and communication interface 930 are implemented independently, the memory 910, processor 920, and communication interface 930 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0162] Optionally, in a specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

[0163] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0164] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0165] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, data subscriber line (DSL)) or wireless (e.g., infrared, Bluetooth, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid-state drive (SSD)). It is worth noting that the computer-readable storage medium mentioned in the present disclosure may be a non-volatile storage medium, in other words, a non-transient storage medium.

[0166] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0167] In the description of the embodiments of the present disclosure, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0168] In the description of the embodiments of the present disclosure, unless otherwise specified, " / " means or. For example, A / B can mean A or B. "And / or" in this document is only a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0169] In the description of the embodiments of the present disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0170] The above description is merely an exemplary embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for automatically labeling a three-dimensional model, comprising: Obtaining, from multiple virtual camera perspectives, mask images of various semantic labels in a composite model to be labeled, where the composite model to be labeled is a three-dimensional model including multiple semantics; Determine component channel graphs of respective sampling point sets of the composite model to be annotated under the perspectives of the multiple virtual cameras; Based on the component channel maps and mask maps under the multiple virtual camera perspectives, labeling the semantic labels to which each component in the composite model to be labeled belongs includes: Voting for the semantic labels to which each component belongs based on the mask images corresponding to the component channel images of each component under the multiple virtual camera perspectives, and obtaining the voting probability that each component belongs to each semantic label; For each component, determine the semantic label to which the component belongs based on the maximum value of the voting probabilities that the component belongs to each semantic label; Among them, the component channel map describes the component to which each sampling point belongs when observing a set of sampling points from the perspective of a virtual camera. The mask map is used to indicate the semantic labels of each part of the composite model to be annotated projected into the two-dimensional space from the perspective of the virtual camera. The voting probability is obtained by integrating the voting results of multiple virtual camera perspectives.

2. The method according to claim 1, wherein The method of obtaining mask images of various semantic labels in the composite model to be labeled under multiple virtual camera perspectives includes: For each virtual camera perspective, perform the following operations: Rendering the composite model to be annotated under the perspective of the virtual camera to obtain a rendering to be processed corresponding to the perspective of the virtual camera; Inputting the rendered image to be processed into an object detection model to obtain object detection results corresponding to each semantic label in the composite model to be annotated; The target detection results are input into the segmentation model to obtain mask images corresponding to each semantic label.

3. The method according to claim 1 or 2, further comprising determining the multiple virtual camera perspectives based on the following method: Taking the target anchor point of the composite model to be annotated as a reference, establishing multiple viewing angle ranges of a virtual camera focusing on the target anchor point; Sampling is performed within the multiple viewing angles to obtain the multiple virtual camera viewing angles.

4. The method according to claim 1, wherein Determining the component channel graphs of the sampling point sets of the composite model to be labeled under the respective component channel graphs of the multiple virtual camera perspectives includes: Sampling the composite model to be annotated to obtain the sampling point set; the sampling point set is used to describe the overall appearance characteristics of the composite model to be annotated in a world coordinate system; For each virtual camera viewpoint, perform the following operations: Mapping the sampling point set to a to-be-processed rendering of the composite model to be annotated under the perspective of the virtual camera to obtain a mapping point set of the sampling point set in the to-be-processed rendering; determining a component label for each sampling point in the set of sampling points based on component information of each mapping point in the set of mapping points in the composite model to be labeled; Based on the component labels of the respective sampling points, a component channel map under the perspective of the virtual camera is generated.

5. The method according to claim 1 or 4, wherein: In the component channel graph, each component corresponds to an image channel, and different components are expressed using different channel values.

6. The method according to claim 1, wherein The component channel graphs of each component are based on the mask graphs corresponding to the multiple virtual camera perspectives, and voting for the semantic labels to which each component belongs to obtain the voting probability that each component belongs to each semantic label, including: Initializing a voting matrix, wherein a first direction in the voting matrix includes multiple semantic labels in the composite model to be annotated, each semantic label corresponds to a respective second direction, each component corresponds to an element in the second direction, and each element is initialized to a default value; Perform the following operations iteratively on the multiple virtual camera perspectives to obtain the voting probability that each component belongs to each semantic label: For each component in the current voting matrix, determine the mask map under the virtual camera perspective to which the component channel map under the virtual camera perspective belongs, and obtain a target mask map of the component; Optimizing the value of a target element in the current voting matrix according to a first strategy so that the value of the target element is higher than the value of the target element in the current voting matrix; the target element is an element of a semantic label corresponding to the target mask image to which the component belongs; The other elements in the current voting matrix other than the target element in the first direction are optimized according to a second strategy, so that the values ​​of the other elements are lower than the values ​​of the other elements in the current voting matrix.

7. The method according to claim 1, further comprising: Extracting components with the same semantic label from the composite model to be annotated; Components with the same semantic label are combined into independent models, and corresponding semantic labels are associated with the independent models.

8. An automatic annotation device for a three-dimensional model, comprising: An acquisition module is used to respectively acquire a mask image of each semantic label in a composite model to be labeled under multiple virtual camera perspectives, wherein the composite model to be labeled is a three-dimensional model including multiple semantics; a determination module, configured to determine component channel maps of the sampling point set of the composite model to be annotated under the respective perspectives of the multiple virtual cameras; A labeling module is used to label the semantic labels of each component in the composite model to be labeled based on the component channel map and mask map under the multiple virtual camera perspectives, including: Voting for the semantic labels to which each component belongs based on the mask images corresponding to the component channel images of each component under the multiple virtual camera perspectives, and obtaining the voting probability that each component belongs to each semantic label; For each component, determine the semantic label to which the component belongs based on the maximum value of the voting probabilities that the component belongs to each semantic label; Among them, the component channel map describes the component to which each sampling point belongs when observing a set of sampling points from the perspective of a virtual camera. The mask map is used to indicate the semantic labels of each part of the composite model to be annotated projected into the two-dimensional space from the perspective of the virtual camera. The voting probability is obtained by integrating the voting results of multiple virtual camera perspectives.

9. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images