Method for three-dimensional structure understanding of large model based on multi-view layered structure clues

Abstract

The application discloses a kind of big model three-dimensional structure understanding method based on multi-view layered structure clue, first to target three-dimensional structure does voxelization processing and spatial hierarchy division, and using interlayer difference visual coding strategy is marked to the structure of each layer divided explicitly;Then, the stereogram, front view, side view and overhead view of target three-dimensional structure are extracted, and the multi-view layered structure clue is constructed by joint display after each layer structure is marked;Finally, according to the multi-view layered structure clue, the structure understanding instruction is constructed for each layer structure in turn, and the big model is guided layer by layer based on instruction to carry out three-dimensional structure understanding.Compared with prior art, the spatial understanding ability of the big model for complex three-dimensional structure is significantly enhanced, and is suitable for any spatial structure analysis scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and large model technology, and in particular to a method for understanding the three-dimensional structure of large models based on multi-view hierarchical structural cues. Background Technology

[0002] 3D structural understanding is a key task in the field of computer vision, aiming to identify and infer the spatial location, hierarchical relationships, and combination methods of the components in a 3D structure. It is widely used in architectural modeling, engineering assembly, industrial design, and other scenarios. Traditional methods rely on computer graphics technologies such as point cloud processing, which, while possessing a certain level of accuracy, have significant limitations in generalization ability and semantic understanding.

[0003] With the rapid development of multimodal large models (MLLMs), such as GPT-4V and Gemini-Pro-Vision, these models have acquired the ability to jointly model graphics and language, demonstrating strong potential in tasks such as text-image question answering and cross-modal reasoning. Utilizing multimodal large models for 3D structural understanding holds promise for breaking through traditional geometric processing paradigms, enabling comprehensive analysis of structural semantics and spatial relationships, and providing superior solutions for applications such as automated structural recognition and structural evaluation.

[0004] However, current multimodal large models still face multiple challenges in understanding the spatial structure of complex 3D structures. First, multimodal large models are mainly designed for 2D image tasks and lack the ability to model hierarchical information in 3D structures, making it difficult to accurately understand the hierarchical relationship between different structural units. Second, without explicit structural guidance, multimodal large models are prone to inference ambiguity, resulting in a lack of controllability and spatial consistency in structural recognition results. Summary of the Invention

[0005] This invention provides a method for understanding the three-dimensional structure of large models based on multi-view hierarchical structural clues, aiming to effectively solve the above-mentioned technical problems.

[0006] According to a first aspect of the present invention, the present invention provides a method for understanding the three-dimensional structure of a large model based on multi-view hierarchical structural cues, characterized in that it is applied to guide a large model to reason about the three-dimensional structure layer by layer, comprising the following steps: Step S1, dividing the target three-dimensional structure into spatial hierarchies; Step S2, using an inter-layer differential visual encoding strategy to explicitly label each layer of structure formed by the spatial hierarchical division; Step S3, constructing multi-view hierarchical structural cues by combining the explicit labels of each layer of structure with several views of the target three-dimensional structure; Step S4, constructing structural understanding instructions for each layer of structure according to the multi-view hierarchical structural cues; Step S5, guiding the large model to reason about the structure of the corresponding layer layer by layer according to the structural understanding instructions to obtain a complete structural understanding.

[0007] Furthermore, the specific process of spatial hierarchy division is as follows:

[0008] The target 3D structure is voxelized and decomposed into several regular cubic units. Based on the structural distribution of the voxelized target 3D structure S, the target 3D structure S is divided into m layers (m≧2) from top to bottom, resulting in a set of layered structures L(S)={L1(S),L2(S),...,L... m (S)}, where Lj(S) represents the j-th layer (1≦j≦m) of the target three-dimensional structure S.

[0009] Furthermore, the specific process of displaying annotations includes:

[0010] From the RGB color space, the R, G, and B color channels are sampled at equal intervals with a fixed step size to generate all possible color combinations, forming a candidate color set C = {c1, c2, ..., c...}. N}, where c i =(R i G i B i ) represents the three-channel values ​​of the candidate color, and N is the total number of all sampled combinations.

[0011] From the candidate color set C, select m colors with large pairwise color differences based on Euclidean distance to form the color set C. * ={c1 * c2 * ,...,c m * Specifically, firstly, color c1 is randomly selected from the candidate color set C. * Add to color set C * Then, for each color c∈C\C * Calculate its relationship with the colored set C based on Euclidean distance. * Find the minimum color distance of the existing colors in C, and select the color with the largest d value to add to C. * Repeat the above steps until m colors are selected. The specific calculation of the minimum color distance is shown in equation (1) below:

[0012]

[0013] Where d(c,C) m ) represents the color c∈C\C * With the color set C * The minimum color distance of an existing color in a given set, R c G c B c R represents the three-channel values ​​of color c.c* G c* B c* Indicates color c * ∈C * The three-channel values.

[0014] Apply color c sequentially to each layer Lj(S)(1≦j≦m) of the voxelized target 3D structure S. j * Apply color.

[0015] Furthermore, the construction process of the multi-perspective hierarchical structural clues is as follows:

[0016] Using orthogonal projection, a stereoscopic image V of the voxelized and colored target 3D structure S is generated from a fixed angle β. 3D (S); Using orthogonal projection, a two-dimensional projection view of the voxelized three-dimensional target structure S is generated from a fixed angle β, including the front view V. front (S), Side view V side (S) and top view V top (S); ultimately forming multi-view layered structural clues for the target three-dimensional structure S:

[0017] M(S)={V 3D (S),V front (S),V side (S),V top (S),L(S),C *}

[0018] In the formula, L(S) is the set of hierarchical structures generated by voxelizing the target 3D structure S, and C * It is a set of colors used to display and annotate the structure of each layer.

[0019] Furthermore, during the generation of several views of the target 3D structure, lighting, shadows, and material effects are turned off, and only the color identifiers corresponding to each structural level are retained.

[0020] Furthermore, each layer of structure L j (S) The corresponding structural understanding instructions must include at least: the current layer L j The numbering and color c of (S)(1≦j≦m) j * The required combination of views, the default is V. 3D (S),V front (S),V side (S),V top(S) The required output structural information dimension, including the arrangement and distribution shape of voxels in this layer, and if j>1, the inference result of the previous layer is added to the current layer's structural understanding instruction to assist the current layer's structural understanding, ultimately forming an instruction set Q(S)={q1,q2|R1,q3|R2,...,q m |R m-1}, where R j (1≦j≦m) represents the reasoning result for the j-th layer.

[0021] Furthermore, the specific process of step S5 is as follows:

[0022] Each layer of structure L j The structural understanding task of (S)(1≦j≦m) uses multi-perspective hierarchical structural cues M(S) and structural understanding instructions q j (1≦j≦m), while introducing the reasoning result R from the previous layer. j-1 (If j>1) Perform inference and output the understanding result R of this layer. j =LLM(M(S),q j ,R j-1 )(1≦j≦m), where R j This represents the structural understanding result of the j-th layer, including the arrangement and distribution shape of voxels in that layer. Finally, the reasoning results of each layer are combined into a complete structural understanding output R(S) = {R1, R2, ..., R...}. m}

[0023] According to a second aspect of the present invention, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the steps of the above-described method for understanding the three-dimensional structure of a large model based on multi-view hierarchical structural cues.

[0024] According to a third aspect of the invention, the invention also provides a storage medium storing a plurality of instructions adapted for loading by a processor to execute the steps of any of the above-described methods for understanding the three-dimensional structure of a large model based on multi-view hierarchical structural cues.

[0025] Through one or more embodiments of the above-described embodiments of the present invention, at least the following technical effects can be achieved: The present invention displays and annotates the structure of each layer of the target 3D structure, establishes multi-view hierarchical structural clues by combining multiple views of the target 3D structure, and constructs structural understanding instructions for each layer. Based on these instructions, the large model is guided to perform structural understanding layer by layer, thereby better assisting the large model in completing the overall structural reasoning of the target 3D structure. Compared with the prior art, the present invention significantly improves the spatial understanding capability of the large model for complex 3D structures. The present invention is applicable to any spatial structure analysis scenario. Attached Figure Description

[0026] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0027] Figure 1 This is a flowchart of the method for understanding the three-dimensional structure of a large model based on multi-view hierarchical structural clues provided in the embodiments of the present invention;

[0028] Figure 2 This is a schematic diagram illustrating the process of structural reasoning for a large-model 3D structure understanding method based on multi-view hierarchical structural clues for a target 3D structure S, as provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0030] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0031] To address the aforementioned technical problems, this invention provides a method for understanding the 3D structure of large models based on multi-view hierarchical structural cues. The method is described below with reference to the accompanying drawings.

[0032] See Figure 1 This invention provides a method for understanding the three-dimensional structure of a large model based on multi-view hierarchical structural cues, comprising the following steps:

[0033] Step S1: Perform voxelization on the target three-dimensional structure S to complete the spatial hierarchy division and obtain several regular cubic units;

[0034] In this embodiment, Figure 2 Taking the target 3D structure shown as an example, based on the structural distribution of the voxelized 3D structure S, S is divided into 3 layers from top to bottom, resulting in a layered structure set L(S) = {L1(S), L2(S), L3(S)}, where L j (S) represents the j-th layer of the three-dimensional structure S (1≦j≦3).

[0035] Step S2: Employ an inter-layer differential visual coding strategy to explicitly label each layer structure formed by spatial hierarchical division.

[0036] In this embodiment, the R, G, and B channels in the RGB color space are sampled at equal intervals with a fixed step size of 255 to generate all possible color combinations, forming a candidate color set C = {(0,0,0),(0,0,255),(0,255,0),(0,255,255),(255,0,0),(255,0,255),(255,255,0),(255,255,0),(255,255,255)}.

[0037] Select three colors with significant pairwise color differences from the candidate color set C above to form the color set C. * ={(255,255,0),(0,255,255),(255,0,255)}.

[0038] Finally, each layer L of the voxelized and colored 3D structure S is processed sequentially. j (S)(1≦j≦3) Use color c j * Apply color. For example... Figure 2 As shown, the first layer uses the color (255,255,0), which is yellow; the second layer uses the color (0,255,255), which is cyan; and the third layer uses the color (255,0,255), which is magenta.

[0039] Step S3: Combine the display annotations of each layer of structure with several views of the target 3D structure to construct a multi-view hierarchical structural clue.

[0040] In this step, orthogonal projection is used to generate a stereoscopic image V of the voxelized and colored 3D structure S from a fixed angle of 45°. 3D (S). Using orthographic projection, a two-dimensional projection view of the voxelized three-dimensional structure S is generated from a fixed angle of 45°, including the front view V. front (S), Side view V side (S) and top view V top (S). In the above generation process, lighting, shadows, and material effects are turned off, and only the color identifiers corresponding to the structural levels are retained. The final multi-view layered structural cues M(S) = {V} of the three-dimensional structure S are formed. 3D (S),V front (S),V side (S),V top (S),L(S),C *}

[0041] Step S4, based on the multi-perspective layered structure clues, sequentially design each layer L. j (S)(1≦j≦3) constructs the structural understanding instruction q j (1≦j≦3).

[0042] See Figure 2 The instruction contains the following elements: (1) the current target layer L j (S) number and color c j * (2) The required combination of views, the default is V. 3D (S),V front (S),V side (S),V top (S); (3) The required output structural information dimensions, including hierarchical relationships and combination methods; (4) If j>1, the reasoning result of the previous layer is added to the current layer's structural understanding instruction to assist in the structural understanding of the current layer. For example, q1 = "Taking the first layer of the three-dimensional structure S as the analysis object, combined with its color identifier c1 * Please use the stereoscopic image V in combination. 3D (S), Front View V front (S), Side view V side (S) and top view V top (S) identifies the arrangement and distribution shape of voxels in the layer and outputs a structured description result.

[0043] The final instruction set is Q(S) = {q1,q2|R1,q3|R2}, where R... j (1≦j≦3) represents the reasoning result for the j-th layer.

[0044] Step S5 guides the large model to reason about the structure of the corresponding layer layer by layer according to the structural understanding instructions, thereby obtaining a complete structural understanding.

[0045] In this embodiment, the large model is invoked sequentially according to hierarchical order for inference. Each layer of structure L j The structural understanding task of (S)(1≦j≦m) uses multi-perspective hierarchical structural cues M(S) and structural understanding instructions q j (1≦j≦m), while introducing the reasoning result R from the previous layer. j-1 (If j>1). Inference output result R j =LLM(M(S),q j ,R j-1 )(1≦j≦m), where R j This represents the structural understanding result of the j-th layer, including the arrangement and distribution shape of voxels in that layer.

[0046] For example, R3 = "The third layer is composed of voxel units, which are arranged in a rectangular shape with a length and width of 3 units and 3 units respectively. They are arranged compactly without gaps, and are judged to be a regular and continuous planar layer."

[0047] Finally, the reasoning results of each layer are combined into a complete structural understanding output R(S) = {R1, R2, ..., R...} m}

[0048] Based on any of the above embodiments, another embodiment of the present invention provides an electronic device, which may include: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor may invoke logical instructions in the memory to execute the above method.

[0049] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] On the other hand, embodiments of the present invention also provide a storage medium storing a plurality of instructions adapted for loading by a processor to execute the large model 3D structure understanding method based on multi-view hierarchical structural cues provided in the above embodiments.

[0051] On the other hand, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0052] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0054] In summary, although the present invention has been disclosed above with reference to preferred embodiments, the above preferred embodiments are not intended to limit the present invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims.

Claims

1. A multi-view hierarchical structure cue-based large model three-dimensional structure understanding method, characterized in that, The application is applied to guiding a large model to infer a three-dimensional structure layer by layer, including the following steps: Step S1, performing spatial hierarchical division on a target three-dimensional structure; Step S2, performing explicit labeling on each layer structure formed by spatial hierarchical division by using an inter-layer differential visual coding strategy; Step S3, constructing multi-view hierarchical structure clues by combining the explicit labeling of each layer structure and several views of the target three-dimensional structure; Step S4, constructing structure understanding instructions for each layer structure according to the multi-view hierarchical structure clues; Step S5, guiding the large model to infer the structure of the corresponding layer layer by layer according to the structure understanding instructions to obtain complete structure understanding; wherein each layer structure L j (S) corresponds to the structure understanding instruction at least contains: the number of the current layer L j (S) and the color c j * , the required view combination of the reference, the required output structure information dimension, including the arrangement and distribution shape of the voxels in the layer, and if j>1, the last layer reasoning result is added to the current layer structure understanding instruction, used to assist the structure understanding of the current layer, and finally forms the instruction set Q(S)={q1, q2|R1, q3|R2,..., q m |R m-1}, wherein R j represents the reasoning result for the jth layer, 1≤j≤m, and m is the total number of layers of the target three-dimensional structure S. The specific process of step S5 is as follows: Each layer structure L j The structure understanding task of (S) uses multi-view hierarchical structure cues M(S) and structure understanding instructions q j At the same time, the inference result of the previous layer is introduced for inference, and the understanding result R of the layer is output j = LLM(M(S), q j , R j-1 ), where R j represents the structure understanding result of the jth layer, including the arrangement and distribution shape of the voxels in the layer, and finally the inference results of each layer structure are combined into the complete structure understanding output R(S) = {R1, R2,..., R m}, and LLM is a large model.

2. The large model three-dimensional structure understanding method based on multi-view layered structure cues according to claim 1, characterized in that, The specific process of the spatial hierarchical division is as follows: Perform voxelization processing on the target three-dimensional structure to decompose it into several regular cubic units; Based on the structural distribution of the voxelized target 3D structure S, the target 3D structure S is divided into m layers from top to bottom, resulting in a layered structure set L(S) = {L1(S), L2(S), ..., L...} m (S)}, where Lj(S) represents the j-th layer of the target three-dimensional structure S.

3. The large model three-dimensional structure understanding method based on multi-view layered structure cues according to claim 1, characterized in that, The specific process of the explicit labeling includes: From the RGB color space, the R, G, and B color channels are sampled at equal intervals with a fixed step size to generate all possible color combinations, forming a candidate color set C={c1, c2, ..., c...} N }, where c i =(R i G i B i ) represents the three-channel values ​​of the candidate color, and N is the total number of all sample combinations; Select m colors with larger color difference from the candidate color set C according to Euclidean distance to form the color assignment set C * ={c1 * , c2 * ,..., c m *} using the color c sequentially for each layer structure of the voxelized target three-dimensional structure S j * is colored.

4. The large model three-dimensional structure understanding method based on multi-view layered structure cues according to claim 1, characterized in that, The construction process of the multi-view hierarchical structure clues is as follows: generating a perspective view V of the voxelized and colored target three-dimensional structure S from a fixed angle using orthogonal projection 3D (S); The two-dimensional projection view of the voxelized target three-dimensional structure S is generated from a fixed angle in an orthogonal projection manner, including the front view V front (S), the side view V side (S), and the top view V top (S); Finally, the multi-view hierarchical structure clues of the target three-dimensional structure S are formed: M(S) = {V 3D (S), V front (S), V side (S), V top (S), L(S), C *} In the formula, L(S) is a layered structure set generated by voxelizing a target three-dimensional structure S, C * is a color set used to display the labeling of each layer structure.

5. The method of claim 4, wherein the method further comprises: In the generation process of the several views of the target three-dimensional structure, the lighting, shadow and material effects are turned off, and only the color identification corresponding to each structure level is retained. 6.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

7. A storage medium, characterized by The storage medium has a plurality of instructions stored therein, and the instructions are adapted to be loaded by the processor to execute the steps of the method in any one of claims 1 to 5.

Images