A voxel generation, exploration and editing method and device based on clip-forge model

Abstract

The present invention discloses a voxel generation exploration and editing method based on the clip‑forge model, including constructing a material library, which includes material images and corresponding voxel vectors, and encoding and clustering based on the voxel vectors to generate a corresponding model cluster set; obtaining a user's query text and / or image samples, and inputting them into a pre-trained voxel generation model to output the corresponding voxel encoding; performing similarity matching based on the voxel encoding and each model cluster subset in the model cluster set to visualize the output of the model cluster subset with the highest matching degree; and editing and adjusting based on the model cluster subset to visualize the output voxel model. The present invention also provides a voxel generation exploration and editing device. The method provided by the present invention can enhance the user's control and creativity in the process of voxel model construction.

Description

Technical Field

[0001] The invention belongs to the technical field of three-dimensional image construction, and in particular relates to a voxel generation exploration and editing method and device based on a clip-forge model. Background Art

[0002] As the basic unit in three-dimensional space, voxels play an important role in many fields. In the field of pixel game industry, voxels provide game designers and players with flexible tools to build and interact with game worlds, allowing the creation of complex environments and terrains, enhancing the playability and creativity of games. In the field of three-dimensional imaging, voxels are the key to volume rendering, enabling professionals to observe and analyze three-dimensional data from multiple angles, and are applied to the construction and modification of models in computer-aided design and manufacturing. In the field of scientific data and medical imaging, voxels form the basis of three-dimensional pixels such as CT and MRI scan images, which are crucial for diagnosis and treatment planning. At the same time, in scientific research, voxels are used to represent and analyze data distribution in three-dimensional space, promoting a deeper understanding of complex phenomena. These applications demonstrate the important value of voxels in capturing and analyzing three-dimensional spatial information. In the field of three-dimensional creation, the complexity of professional tools often limits the workflow of voxel creators, making it only users with professional skills to create models efficiently.

[0003] Patent document CN118397198A discloses a three-dimensional reconstruction method for pathological image feature analysis, including: obtaining a stained image of each slice level, and then obtaining each initial three-dimensional voxel block; determining the degree of grayscale distribution disorder of the area to be analyzed for each pixel point in the stained image, and determining the matching degree of each pair of matching pixels in the stained images of two adjacent slice levels; determining the division condition index of each initial three-dimensional voxel block according to the matching degree and the grayscale value of each pixel point in the stained image of each slice level in each initial three-dimensional voxel block, performing threshold judgment, obtaining each final three-dimensional voxel block, constructing a three-dimensional model, and obtaining a reconstructed three-dimensional model of tumor tissue.

[0004] Patent document CN118505936A discloses a 3D model building block intelligent conversion and construction system, including the following steps: Step S1: Import a 3D model, obtain the feature data of the 3D model, preset data set by the user, and target building block parameters; Step S2: Generate a bounding box that can cover the 3D model and rasterize the 3D model to form a number of cells, traverse all cells to obtain the position coordinates of the cells, and establish a cell data set for each cell according to the position coordinates; Step S3: Optimize the number of cells in the cell data set, filter out cells that do not participate in merging, randomly shuffle the remaining cells after optimization and merge them, so that the merged cell features can correspond to the target building block parameters; Step S4: Retrieve the corresponding color, texture and coordinate parameters of the merged cells, and render the generated model. Summary of the invention

[0005] The purpose of the present invention is to provide a voxel generation exploration and editing method and device based on the clip-forge model, which can enhance the user's control and creativity in the voxel model construction process.

[0006] In order to realize the first object of the present invention, the following technical solution is provided: a voxel generation exploration and editing method based on the clip-forge model, comprising the following steps:

[0007] Constructing a material library, which includes material images and corresponding voxel vectors, and encoding and clustering based on the voxel vectors to generate a corresponding model cluster set;

[0008] Obtain the user's query text and / or image sample and input it into the pre-trained voxel generation model to output the corresponding voxel encoding. The generation model is built based on the clip model framework, which includes an image generation module, an encoding module, a feature extraction module, and a fusion module:

[0009] The image generation module generates a corresponding image according to the input text;

[0010] The encoding module encodes the voxel vector corresponding to the image generated by the image generation module or / and the input image to obtain an initial voxel code;

[0011] The feature extraction module is used to extract the spatial features in the initial voxel encoding;

[0012] The fusion module fuses the spatial feature with the initial voxel code to output a voxel code for similarity matching;

[0013] Based on the voxel encoding, similarity matching is performed with each model cluster subset in the model cluster set, so as to visually output the model cluster subset with the highest matching degree;

[0014] Editing and adjusting are performed based on the subset of model clusters to visualize the output voxel model.

[0015] By encoding the voxel data in the ShapeNet database and then performing dimensionality reduction and hierarchical clustering processing to obtain a visual analysis view, an interactive interface is provided to allow users to generate voxel models using text, images, etc., and explore the clustering information of similar generated results in the database, thereby mining the semantic similarity between voxels.

[0016] Specifically, the material library adopts the ShapeNet dataset.

[0017] Specifically, encoding is performed based on the resolution in the voxel vector.

[0018] Specifically, the T-SNE algorithm is used in the encoding process to reduce the dimension of the encoding result.

[0019] Specifically, the clustering adopts a hierarchical clustering method to display different materials in a hierarchical manner.

[0020] Specifically, the similarity matching is based on the position information in the dimensional space corresponding to the voxel encoding, and uses the Ward squared sum of variance algorithm to measure the distance between the voxels and the subsets in the model cluster set.

[0021] In order to achieve the second object of the present invention, the following technical solution is provided: a voxel generation exploration and editing device, which is implemented by the above-mentioned voxel generation exploration and editing method based on the clip-forge model, comprises an input unit, a material exploration unit, an adjustment editing unit and a visualization unit;

[0022] The input unit is used to obtain the query text and / or image sample provided by the user;

[0023] The material exploration unit matches the corresponding model cluster subset according to the input query text and / or image sample;

[0024] The adjustment and editing unit includes a frame selection state, a copy state and a single-side voxel addition state, so as to edit and adjust the model;

[0025] The visualization unit is used to visualize the model cluster subset obtained through exploration and each operation result in the adjustment and editing unit.

[0026] Specifically, each adjustment and editing in the adjustment and editing unit requires a failed voxel detection, and the failed voxel detection matches the model voxels before the operation and the three-dimensional coordinates of the voxels to be added or deleted through a pre-built mapping table to ensure that there is only one voxel at one coordinate in the space.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Through the voxel generation exploration and editing device, users can manually adjust the generation results after exploring the model encoding semantics. The adjusted results can also be encoded as back-end model input. In this way, users can further explore the relationship between voxel space and encoding, thereby providing users with a comprehensive, intuitive and powerful voxel editing and exploration platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart of a voxel generation exploration and editing method based on the clip-forge model provided in this embodiment;

[0030] Figure 2 A schematic diagram showing the hierarchy of each subset in the model cluster set provided in this embodiment;

[0031] Figure 3 A workflow diagram of the generation model provided for this embodiment;

[0032] Figure 4 A schematic diagram of the operation flow of the adjustment editing unit provided in this embodiment;

[0033] Figure 5 A schematic diagram of the process of detecting failed voxels provided in this embodiment. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, the voxel generation exploration and editing method based on the clip-forge model provided in this embodiment includes the following steps:

[0036] A material library is constructed, which includes material images and corresponding voxel vectors, and encoding and clustering are performed based on the voxel vectors to generate a corresponding model cluster set.

[0037] The user's query text and / or image samples are obtained and input into the pre-trained voxel generation model to output the corresponding voxel encoding.

[0038] Based on the voxel encoding, similarity matching is performed with each model cluster subset in the model cluster set to visualize and output the model cluster subset with the highest matching degree.

[0039] Editing and adjusting are performed based on the subset of model clusters to visualize the output voxel model.

[0040] Furthermore, in this embodiment, the 32*32*32 voxels in each data are used as the input of the autoencoder to obtain the 128-bit code corresponding to the voxel, and then the code is subjected to T-SNE dimensionality reduction and hierarchical clustering for front-end visualization. Through this process, based on the position of each voxel in the 128-dimensional space, the Ward squared sum of variance algorithm can be used to measure the distance between the voxels and their clusters, and then the semantic similarity between the voxels can be mined. This method not only improves the explorability of voxel data, but also provides a new perspective for understanding the relationship between different voxel features. By exploring these codes, users can gain insight into the distribution pattern of voxels in high-dimensional space, thereby promoting a deep understanding of the semantics of three-dimensional shapes.

[0041] like Figure 2 As shown, the hierarchical display of each subset in the model cluster set provided by this embodiment, the user interface provides the function of in-depth exploration of the sub-clusters and parent clusters of the matching cluster. In addition, the voxel vector editing view allows the user to perform arithmetic operations and combinations on the 128-bit encoded voxels, enabling the user to deeply study and apply the local features of different voxels to achieve feature extraction and innovative combination.

[0042] like Figure 3 The generation model provided in this embodiment includes an image generation module, an encoding module, a feature extraction module and a fusion module:

[0043] The image generation module generates a corresponding image according to the input text.

[0044] The encoding module encodes the voxel vector corresponding to the image generated by the image generation module or / and the input image to obtain an initial voxel code.

[0045] The feature extraction module is used to extract the spatial features in the initial voxel encoding.

[0046] The fusion module fuses the spatial feature with the initial voxel code to output a voxel code for similarity matching.

[0047] That is: it allows users to perform arithmetical combinations on the codes of different voxel models, extract the spatial features of voxels from the initial voxel codes, and based on the vector operations of the word2vec model in natural language processing, for example, "King - Man + Woman = Queen", specifically, users can try to perform arithmetical combinations on the voxel codes of "car" and "plane", and after normalization, input them into the Clip-Forge model to generate a car model with airplane wing features. These operations help users to extract and fuse specific features more effectively. In this way, it is proved that on the basis of using the CLIP model for text feature extraction, the encoding results obtained by combining the autoencoder can not only maintain semantic information, but also show good semantic relationships when performing feature fusion. The voxel coding editing function of this device not only enriches the user's editing methods for voxel models, but also provides an innovative method for exploring the semantic associations between voxel models. Through arithmetic combination and feature extraction, users can achieve more complex and creative model design in three-dimensional space.

[0048] This embodiment also provides a voxel generation exploration and editing device, which is implemented by the voxel generation exploration and editing method based on the clip-forge model provided in the above embodiment, including an input unit, a material exploration unit, an adjustment editing unit and a visualization unit.

[0049] The input unit is used to obtain the query text and / or image sample provided by the user.

[0050] The material exploration unit matches the corresponding model cluster subset according to the input query text and / or image sample.

[0051] The adjustment and editing unit includes a frame selection state, a copy state and a single-sided voxel addition state, so as to edit and adjust the model.

[0052] The visualization unit is used to visualize the model cluster subset obtained through exploration and each operation result in the adjustment and editing unit.

[0053] Furthermore, if Figure 4 As shown, it is a workflow diagram of adjusting the editing unit in the device provided in this embodiment.

[0054] In the process of adjustment and editing, based on the initial editing state, by clicking to select the voxel model to be edited, you will enter the box selection state. At this time, you can delete the box selected voxel, rotate and / or move, copy the state, and add voxel state on one side. When the box selection is cancelled, an invalid voxel detection is performed;

[0055] When the copy operation is performed after the selection state, the selected voxels are copied to the target position by copying and pasting. When the operation is completed, a failed voxel detection is required to determine whether there is a problem of voxel model overlap;

[0056] When the operation of adding voxel state on one side is performed after the box selection state, the new voxel state is copied to the target position. When the operation is completed, a failed voxel detection is required to determine whether there is a problem of voxel model overlap.

[0057] like Figure 5 As shown, the specific process of the failed voxel detection provided by this embodiment is to maintain a mapping table from three-dimensional coordinates to voxel objects, which is used to reflect the occupancy status of voxel blocks in the current space in real time. For example, the (x, y, z) coordinates of a voxel will be mapped to PositionTable[x * 64 * 64 + y *64 +z], and vice versa. In this way, it is possible to quickly query whether there is already a voxel at the original position after a new voxel is generated at a certain coordinate, thereby ensuring that there is only one voxel object at a coordinate in the space to avoid overlapping occupancy performance of voxel models.

[0058] That is, after each move or addition of voxels, the device will perform conflict detection to ensure that the voxels at the current position will not conflict with the voxels at the original position. If there is a conflict, it must be resolved in time. This mechanism not only ensures the accuracy of voxel editing, but also improves the efficiency of users going back to the previous state when performing an undo operation.

[0059] In summary, the embodiment provides an interactive analysis method for text-based voxel model generation, which enables users to explore the similarities and semantic relationships between models in the encoding space of 3D models.

[0060] At the same time, the user can explore the entire ShapeNet database collection in a hierarchical clustering level display;

[0061] Furthermore, by integrating the generation and editing capabilities, the edited model can also be fed back to the generation device for the user to analyze the properties of its encoding space.

[0062] The technical solution provided by this embodiment is to free users from the high threshold and cumbersome manual operations of traditional voxel editors, thereby simplifying the model generation and editing process. Next, two cases will be used to demonstrate this process.

[0063] Case 1: Assume that the user just wants to obtain a car model, including the following steps:

[0064] Step s11: The user inputs the prompt word "a car".

[0065] Step s12: The backend will generate a corresponding car model without any characteristics according to the prompt word, and search and match similar models in the ShapeNet database. This step will screen out different car models that belong to the category of cars but have different appearances, such as sports cars, racing cars or jeeps.

[0066] Step s13: The front-end device makes an integrated display of similar models. Since the ShapeNet database has been clustered in preprocessing, the front-end will display the voxel models under the four clusters with the highest similarity to the generated voxel model.

[0067] Step s14: The user can click on the image of the cluster to continue exploring similar models for the four subclusters with the highest similarity under the cluster, thereby obtaining more diverse model options.

[0068] Case 2: Assume that the user needs to obtain a model with combined features, such as a car that can fly. The user will enter "a car with wings", but the user will be disappointed to find that the generated result is not as expected. This is because the model cannot recognize text with cross-category features due to the limitation of the training data set. The corresponding technical problem is solved through the feature extraction module and the fusion module, including the following steps:

[0069] Step s21: The user inputs the prompt words "a car" and "a plane" and then generates and filters voxel models of the car and the plane. This step can be considered to include s11 to s14.

[0070] Step s22: The device uses an autoencoder to extract spatial features of the model selected by the user to obtain feature codes of the two models.

[0071] Step s23: The user drags the model generated in s21 from the generation record column to the visual node editing diagram of the fusion module. The device will generate a separate node for each model. The node includes an input interface and an output interface, and the node will be displayed at the same time.

[0072] Step s24: The user drags a multiplication node "Multiply" and a constant node "Constant" to the node editing diagram, enters 0.3 in the input box of the constant node, and connects the output interfaces of the constant node and the aircraft model node to the two input interfaces of the multiplication node respectively, and performs similar operations on the car node, except that the constant node value of the car node is 0.9. Since we need to keep the car as the main body and the aircraft only needs the wing features, the scale factor of the car model is three times that of the aircraft model.

[0073] Step s25: The user drags the addition node "Add" and an empty model node to the node editing diagram, and connects the calculation results of the two multiplication nodes from the output interface to the input interface of the addition node. Finally, the output interface of the addition node is connected to the input interface of the empty model node to obtain a rough car model with wing features after fusion calculation.

[0074] In addition, the terms "upper", "lower", "inner", "outer", "front", "rear" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps described in these embodiments do not limit the scope of the present invention.

[0075] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of implementation of the present invention. All equivalent changes or modifications made according to the structure, characteristics and principles described in the patent application scope of the present invention should be included in the patent application scope of the present invention.

[0076] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A voxel generation exploration and editing method based on the clip-forge model, characterized in that: The following steps are involved: Constructing a material library, which includes material images and corresponding voxel vectors, and encoding and clustering based on the voxel vectors to generate a corresponding model cluster set; Obtain the user's query text and / or image sample and input it into the pre-trained voxel generation model to output the corresponding voxel encoding. The generation model is built based on the clip model framework, which includes an image generation module, an encoding module, a feature extraction module, and a fusion module: The image generation module generates a corresponding image according to the input text; The encoding module encodes the voxel vector corresponding to the image generated by the image generation module or / and the input image to obtain an initial voxel code; The feature extraction module is used to extract the spatial features in the initial voxel encoding; The fusion module fuses the spatial feature with the initial voxel code to output a voxel code for similarity matching; Based on the voxel encoding, similarity matching is performed with each model cluster subset in the model cluster set, so as to visually output the model cluster subset with the highest matching degree; Editing and adjusting are performed based on the subset of model clusters to visualize the output voxel model.

2. The voxel generation exploration and editing method based on the clip-forge model according to claim 1, characterized in that: The material library adopts the ShapeNet dataset.

3. The voxel generation exploration and editing method based on the clip-forge model according to claim 1, characterized in that: Encoding is based on the resolution in the voxel vector.

4. According to the voxel generation exploration and editing method based on the clip-forge model according to claim 1 or 3, the T-SNE algorithm is used to reduce the dimension of the encoding result during the encoding process.

5. The voxel generation exploration and editing method based on the clip-forge model according to claim 1, characterized in that: The clustering adopts a hierarchical clustering method to display different materials in a hierarchical manner.

6. The voxel generation exploration and editing method based on the clip-forge model according to claim 1, characterized in that: The similarity matching is based on the position information in the dimensional space corresponding to the voxel encoding, and uses the Ward square sum of variance algorithm to measure the distance between the voxels and the subsets in the model cluster set.

7. A voxel generation exploration and editing device, characterized in that: It is implemented by the voxel generation exploration and editing method based on the clip-forge model as described in any one of claims 1 to 6, comprising an input unit, a material exploration unit, an adjustment editing unit and a visualization unit; The input unit is used to obtain the query text and / or image sample provided by the user; The material exploration unit matches the corresponding model cluster subset according to the input query text and / or image sample; The adjustment and editing unit includes a frame selection state, a copy state and a single-side voxel addition state, so as to edit and adjust the model; The visualization unit is used to visualize the model cluster subset obtained through exploration and each operation result in the adjustment and editing unit.

8. The voxel generation exploration and editing device according to claim 7, characterized in that: Each time adjustment and editing is performed in the adjustment and editing unit, a failed voxel detection needs to be performed. The failed voxel detection matches the model voxels before the operation and the three-dimensional coordinates of the voxels to be added or deleted through a pre-constructed mapping table to ensure that there is only one voxel at one coordinate in the space.

Images