3D model file storage method, 3D model file retrieval method and 3D model file storage equipment

Abstract

The invention provides a 3D model file storage method, a 3D model file retrieval method and 3D model file retrieval equipment, and the scheme does not only store a 3D model file and corresponding basic information into a database, but converts the 3D model file into a 2D image, and analyzes the 2D image to extract corresponding label information and cross-modal feature vectors, so that the 3D model file is retrieved. According to the method and the system, the 3D model file is mapped into the 3D model file, the 3D model file is jointly written into the database as an additional index mapped to the 3D model file, and the retrieval of the 3D model file is realized by adopting a 2D picture retrieval technology during retrieval, so that the efficiency and the accuracy of retrieval are effectively improved, and efficient retrieval of 3D assets is realized.

Description

Technical Field

[0001] This application relates to the technical field of image retrieval, and particularly to a method and device for storing 3D model files into a database and a retrieval method. Background Art

[0002] With the development of multimedia technology, some companies engaged in creative visual design and production have accumulated a large number of high-quality 3D (Three Dimensions) assets during their daily operations. For example, stage designs and props for some TV shows. In the early days, no corresponding 3D asset databases were established for these 3D assets. Basically, they were all stored by individual creators in specific fixed storage devices such as hard disks and not uniformly managed. Therefore, it is necessary to establish a 3D asset database to store them. However, when establishing a 3D asset database, problems such as how to efficiently store the 3D model files corresponding to these 3D assets and how to achieve efficient retrieval after storage will be faced.

[0003] Currently, there is no efficient storage and retrieval solution for 3D assets in the industry. Summary of the Invention

[0004] An object of this application is to provide a method and device for storing 3D model files into a database and a retrieval method.

[0005] To achieve the above object, an embodiment of this application provides a method for storing 3D model files into a database, and the storage method includes:

[0006] Obtain a 3D model file to be stored into the database and its corresponding basic information;

[0007] Preprocess the 3D model file to obtain 2D images of at least one view of the 3D model file;

[0008] Perform object detection on the 2D images, identify the objects included in the 2D images, and generate tag information based on the objects included in the 2D images;

[0009] Extract cross-modal feature vectors from the 2D images;

[0010] Write the 3D model file, its corresponding 2D images, basic information, tag information, and cross-modal feature vectors into a database.

[0011] Further, preprocessing the 3D model file to obtain 2D images of at least one view of the 3D model file includes:

[0012] Preprocess the 3D model file to obtain 2D pictures of multiple key perspectives of the 3D model file, where the key perspectives include the front view, side view, and top view.

[0013] Further, perform cross-modal feature extraction on the 2D pictures to obtain cross-modal feature vectors, including:

[0014] Use the CLIP model to perform cross-modal feature extraction on the 2D pictures to obtain cross-modal feature vectors for representing the image features and text description features of the 2D pictures in the same vector space.

[0015] Further, before obtaining the 3D model file to be stored in the database and its corresponding basic information, it also includes:

[0016] Obtain the user input content;

[0017] Judge the user's intention according to the user input content;

[0018] If it is determined that the user's intention is to store the 3D model file in the database, prompt the user to input the 3D model file to be stored in the database and its corresponding basic information.

[0019] The embodiment of the present application also provides a 3D model file retrieval method. The 3D model file is written into the database by using the method described in any one of claims 1 to 4. The retrieval method includes:

[0020] Obtain the query information input by the user;

[0021] Determine the retrieval method according to the query information;

[0022] According to the corresponding retrieval method, use the query information to match in the database to obtain a matching result;

[0023] Determine the corresponding 3D model file according to the matching result as the retrieval result of this retrieval.

[0024] Further, the retrieval method at least includes any one of the following: basic information retrieval, target object retrieval, cross-modal feature retrieval;

[0025] According to the corresponding retrieval method, use the query information to match in the database to obtain a matching result, including:

[0026] According to the corresponding retrieval method, use the query information to match with the basic information, label information, and / or cross-modal feature vectors in the database to obtain a matching result.

[0027] Furthermore, when the search method is a target object search, the query information is used to perform a match in the database according to the corresponding search method to obtain a matching result, including:

[0028] Constructing a first set according to the query tag about the target object in the query information;

[0029] Using a semantic matching algorithm, semantically matching the first set with the second set to determine the semantic similarity between the first set and the second set, wherein the second set is constructed by label information corresponding to any one of the 3D model files in the database;

[0030] If the semantic similarity exceeds a similarity threshold, it is determined that the second set matches the first set, and the 3D assets corresponding to the second set are used as the current matching result.

[0031] Furthermore, before obtaining the query information input by the user, the following is also included:

[0032] Get user input;

[0033] Determining user intention based on the user input content;

[0034] If it is determined that the user's intention is to retrieve 3D model files, the user is prompted to enter query information.

[0035] Furthermore, the query information input by the user is obtained, including:

[0036] Use a large language model to obtain the query information entered by the user through multiple rounds of dialogue with the user.

[0037] Furthermore, according to the matching result, a corresponding 3D model file is determined as a search result of this search, including:

[0038] When there are multiple matching results, sorting the matching results;

[0039] According to the sorted matching results, a 3D model file corresponding to at least one matching result ranked top is selected as the search result of this search.

[0040] An embodiment of the present application also provides a computing device, which includes a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the 3D model file warehousing method or the 3D model file retrieval method.

[0041] The embodiments of the present application also provide a computer-readable medium, on which computer program instructions are stored, and the computer-readable instructions can be executed by a processor to implement the 3D model file warehousing method or the 3D model file retrieval method.

[0042] Compared with the prior art, a 3D model file warehousing solution provided by the embodiments of the present application first obtains a 3D model file to be warehoused and corresponding basic information, then preprocesses the 3D model file to obtain 2D pictures of at least one perspective of the 3D model file, performs object detection on the 2D pictures, identifies the objects included in the 2D pictures, generates label information according to the objects included in the 2D pictures, then performs cross-modal feature extraction on the 2D pictures to obtain cross-modal feature vectors, and writes the 3D model file, its corresponding 2D pictures, basic information, label information, and cross-modal feature vectors into a database. This solution does not simply store the 3D model file and its corresponding basic information in the database, but first converts the 3D model file into 2D pictures, analyzes the 2D images to extract the corresponding label information and cross-modal feature vectors, which are used as additional indexes mapped to the 3D model file and written into the database together, thereby realizing the efficient warehousing of 3D assets and facilitating users to quickly retrieve the required 3D model files from the database.

[0043] In addition, the embodiments of the present application also provide a 3D model file retrieval solution on this basis. After the 3D model file in this solution is written into the database by using the foregoing warehousing method, the query information input by the user can be obtained, then according to the query information, the retrieval method is determined, and according to the corresponding retrieval method, the query information is used to match in the database to obtain a matching result. Finally, according to the matching result, the corresponding 3D model file is determined as the retrieval result of this retrieval. In the retrieval solution, the retrieval technology of 2D pictures is used to implement the retrieval of 3D model files, and according to different query information input by the user, a suitable retrieval method can be adopted to match in the database, thereby effectively improving the efficiency and accuracy of retrieval and realizing the efficient retrieval of 3D assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present application will become more obvious:

[0045] Figure 1 It is a processing flow chart of a 3D model file warehousing method provided by the embodiments of the present application;

[0046] Figure 2 It is a processing flow chart of a 3D model file retrieval method provided by the embodiments of the present application;

[0047] Figure 3 This is a specific implementation process for implementing target object retrieval using the corresponding retrieval algorithm in the embodiments of the present application;

[0048] Figure 4 This is a processing flow chart for implementing the warehousing of 3D model files using a solution provided by the embodiments of the present application;

[0049] Figure 5 This is a processing flow for implementing the retrieval of 3D model files using a solution provided by the embodiments of the present application;

[0050] Figure 6 This is a schematic diagram of the overall architecture of a 3D asset database system implemented by the solution of the embodiments of the present application;

[0051] Like reference numerals in the drawings represent like or similar components. Detailed Description of the Embodiments

[0052] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0053] Those skilled in the art of the present technology can understand that unless specifically stated otherwise, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any unit and all combinations of one or more related listed items.

[0054] A 3D model file storage method and a retrieval method provided by an embodiment of the present application. When storing a 3D model file, not only the 3D model file and its corresponding basic information are stored in the database, but first the 3D model file is converted into a 2D image, and the corresponding tag information and cross-modal feature vectors are extracted by analyzing the 2D image, which are used as additional indexes mapped to the 3D model file and written into the database together. Thus, the efficient storage of 3D assets is realized, which is convenient for users to quickly retrieve the required 3D model files from the database. Correspondingly, when retrieving a 3D model file, the retrieval technology of 2D images can be used to realize the retrieval of 3D model files, and according to different query information input by users, appropriate retrieval methods can be adopted to perform matching in the database, thereby effectively improving the efficiency and accuracy of retrieval and realizing the efficient retrieval of 3D assets.

[0055] In an actual scenario, the execution subject of this method can be a user device, a network device, or a device formed by integrating a user device and a network device through a network, or it can also be an application program running on the above devices. The user device includes but is not limited to various terminal devices such as computers, mobile phones, and tablets; the network device includes but is not limited to network hosts, single network servers, servers centralized with multiple network servers, or servers in a distributed cloud network. Here, the distributed cloud network is composed of a large number of hosts or network servers based on cloud computing.

[0056] Figure 1 The processing flow of a 3D model file storage method provided by an embodiment of the present application is shown, which at least includes the following processing steps:

[0057] Step S101, obtain the 3D model file to be stored and the corresponding basic information. Among them, the 3D model file refers to various 3D assets that need to be stored and managed by the storage method in this solution, and the basic information refers to a part of the relevant information about the 3D model file, such as it can include but is not limited to file name, file size, file scene location, file description, author, version information, etc.

[0058] Step S102: Preprocess the 3D model file to obtain 2D (Two Dimensions) pictures of at least one view of the 3D model file. Among them, the preprocessing mainly refers to using the 3D model file to export views of different perspectives, so as to obtain corresponding 2D pictures. In this embodiment, 2D pictures of the most representative key perspectives can be exported, that is, preprocess the 3D model file to obtain 2D pictures of multiple key perspectives of the 3D model file, where the key perspectives include front view, side view, top view, etc. In an actual scenario, the preprocessing of exporting views can be completed using various 3D model software. For example, 3D model software such as 3ds Max, Blender, and Cinema 4D all have similar functions.

[0059] Step S103: Perform object detection on the 2D pictures, identify the objects included in the 2D pictures, and generate label information according to the objects included in the 2D pictures. Taking a certain 2D picture as an example, if the 2D picture is the front view of a 3D model file of a stage design, and the picture includes objects such as a stage, a flag, and flowers, then through object detection, it can be recognized that the above objects are included in the 2D picture, and then corresponding label information can be generated according to these objects. Specifically, the label information can correspond to the recognized objects. For example, the labels generated from the above 2D picture can at least include: "stage", "flag", "flower".

[0060] Step S104: Extract cross-modal features from the 2D pictures to obtain cross-modal feature vectors. Among them, the cross-modal feature vectors can represent the image features and text description features of the 2D pictures in the same vector space, so as to facilitate cross-modal feature retrieval such as "text-to-image search" or "image-to-text search".

[0061] In some embodiments of the present application, when extracting cross-modal features from the 2D pictures, the CLIP (Contrastive Language-Image Pretraining) model can be specifically used. Thus, in the solution of this embodiment, the CLIP model can be used to extract cross-modal features from the 2D pictures to obtain cross-modal feature vectors for representing the image features and text description features of the 2D pictures in the same vector space.

[0062] Among them, the CLIP model is a multi-modal deep learning model designed to jointly pre-train natural language tasks and image understanding tasks through contrastive learning, thereby achieving cross-modal information interaction and fusion. The core idea of CLIP is to encode images and texts into the same vector space, so that in this space, the representations of relevant images and texts can be close to each other, while those of irrelevant ones are far apart. This design not only simplifies the complexity of cross-modal interaction but also greatly improves the model's performance in various tasks. The CLIP model mainly consists of two independent encoders:

[0063] Image encoder: Responsible for embedding images into the semantic space. Commonly used image encoders include convolutional neural networks (such as ResNet) and Transformer models (such as Vision Transformer, ViT). These encoders can extract high-dimensional features from images and convert them into fixed-size vector representations.

[0064] Text encoder: Responsible for embedding texts into the same semantic space as images. Text encoders usually adopt Transformer-based structures, which can handle long-range dependencies in natural language and effectively capture semantic information in texts. After the text input passes through the embedding layer and position encoding, it enters multiple layers of Transformer modules, and finally generates the feature vector of the text.

[0065] During the training process, the CLIP model is pre-trained using a large-scale image-text pair dataset. These datasets contain rich images and corresponding descriptive texts, providing extensive learning materials for the model. The goal of the model is to make the feature vectors of matching image-text pairs as close as possible and those of non-matching image-text pairs as far apart as possible through contrastive learning. This is achieved through a contrastive loss function (such as the InfoNCE loss), which is used to optimize the distance between the feature vectors of the same pair of image and text to make them closer.

[0066] Therefore, the above technical features enable the CLIP model to achieve the alignment of images and texts in the vector space through contrastive learning, thereby efficiently and accurately obtaining cross-modal feature vectors that can represent the image features and text description features of the 2D pictures in the same vector space in this solution.

[0067] Step S105, write the 3D model file, its corresponding 2D picture, basic information, label information, and cross-modal feature vector into the database.

[0068] After obtaining the 2D image, basic information, label information, and cross-modal feature vectors corresponding to the 3D model file, the 3D model file and its corresponding 2D image, basic information, label information, and cross-modal feature vectors can be jointly written into the database, thus completing the warehousing of 3D assets.

[0069] Since this solution does not simply store the 3D model file and its corresponding basic information in the database, but first converts the 3D model file into a 2D image, analyzes the 2D image to extract the corresponding label information and cross-modal feature vectors, which are used as additional indexes mapped to the 3D model file, and jointly writes them into the database, thus realizing the efficient warehousing of 3D assets and facilitating users to quickly retrieve the required 3D model files from the database.

[0070] In addition, in some embodiments of this application, before obtaining the 3D model file to be warehoused and its corresponding basic information, user input content can be obtained, and then the user intention can be judged according to the user input content. If it is determined that the user intention is to warehouse the 3D model file, the user is prompted to input the 3D model file to be warehoused and its corresponding basic information.

[0071] In an actual scenario, the currently widely used GUI (Graphical User Interface) can be adopted to guide the user to complete the input. After obtaining the input content, the LLM (Large Language Model) can be used for interaction to let the large language model check the user's input content, thereby completing the intention judgment of the user input content, and the user can be guided to input the 3D model file to be warehoused and its corresponding basic information, etc. through dialogue, thus effectively improving the user's interaction experience.

[0072] Based on the above 3D model file warehousing method, an embodiment of this application also provides a 3D model file retrieval method. The 3D model file in this method has been written into the database by using the aforementioned 3D model file warehousing method, and its specific processing flow is as Figure 2 shown, and at least includes the following processing steps:

[0073] Step S201, obtain the query information input by the user. Among them, the query information includes the specific instruction content for retrieving a specific 3D model file. For example, the query information in an actual scenario can include: "Query the model made for XX Theater from December 10th to 20th, using the basic information retrieval mode", "Query the model containing the podium, large speakers, and Chinese knots, according to the target object retrieval mode", "Find the scenes related to the New Year, using the cross-modal feature retrieval mode", etc.

[0074] In some embodiments of the present application, before obtaining the query information input by the user, the content input by the user can be obtained, and then the user's intention can be judged according to the content input by the user. If it is determined that the user's intention is a certain file retrieval, the user will be prompted and required to input relevant query information.

[0075] In this processing process, the currently widely used GUI can also be used to guide the user to complete the input, and the LLM can be used for interaction. The large language model can check the content input by the user, judge according to the user's retrieval intention, and guide the user to input the required query information in a dialog way, so as to improve the user's interaction experience. Further, in the embodiments of the present application, the large language model can also be used to have multiple rounds of conversations with the user, so as to more accurately understand the user's retrieval intention, improve the retrieval accuracy, and provide relevant recommendation information when necessary to help the user refine the query information.

[0076] Step S202: Determine the retrieval method according to the query information.

[0077] Step S203: Use the query information to match in the database according to the corresponding retrieval method, and obtain the matching result.

[0078] In the solution of the embodiment of the present application, at least three retrieval methods are provided, including basic information retrieval, target object retrieval, and cross-modal feature retrieval. Thus, when performing retrieval, the query information can be used to match with the basic information, label information, and / or cross-modal feature vectors in the database according to the corresponding retrieval method to obtain the matching result.

[0079] In an actual scenario, when the user wants to retrieve a 3D model file, they will have a conversation with the large language model. The large language model will determine the retrieval method according to the user's query information, and perform retrieval according to the retrieval method required by the user based on pre-training and the content input by the user. The implementation of this part of the processing can be considered to be achieved by using an agent. The agent can analyze the intention input by the user, and then call different program modules to complete different retrieval tasks.

[0080] Among them, the three retrieval methods are specifically as follows:

[0081] 1. Basic information retrieval.

[0082] When a user is having a conversation with a large language model, they can input basic information, including but not limited to file name, file size, file scene location, file description, file author, version, etc., and specify that the large language model perform a retrieval according to the basic information. At this time, the large language model will extract the query information input by the user and perform a matching query in the database of 3D model files according to the query information. For example, the query information is: "Query the models produced by XX Theater from December 10th to 20th, using the basic information retrieval mode", then the large language model can determine that the retrieval method is basic information retrieval, and then based on the two basic information items of the file generation time (between December 10th and 20th) and the file scene location (file scene location), match with the basic information stored in the database, and thus the corresponding matching results can be obtained.

[0083] 2. Target object retrieval.

[0084] When a user is having a conversation with a large language model and specifies that the large language model perform a query according to a target object, the label information corresponding to the target object will be extracted from the query information input by the user, and a matching query will be performed in the database of 3D model files according to the label information corresponding to these target objects. For example, the query information is: "Query the models containing the rostrum, large speakers, and Chinese knots, according to the target object retrieval mode", the large language model first identifies the vocabulary about the target object, including the rostrum, speakers, and Chinese knots, and thus generates the corresponding label information, such as "rostrum", "speaker", "Chinese knot". Then, based on these label information, match with the label information stored in the database. If the similarity with the label information of a 3D model file exceeds the preset similarity threshold, it can be considered that the two are matched, and the corresponding 3D asset can be obtained.

[0085] In an actual scenario, target object retrieval can be implemented using some off-the-shelf commercial search engines. For example, by connecting to the search engine Elastic Search, the processing process of target object retrieval can be implemented. In some other embodiments of the present application, an efficient retrieval algorithm is also provided to implement the processing of target object retrieval. When implementing target object retrieval using this retrieval algorithm, first, a first set can be constructed according to the query label about the target object in the query information. Then, a semantic matching algorithm is used to semantically match the first set with a second set to determine the semantic similarity between the first set and the second set. Wherein, the second set is constructed from the label information corresponding to any 3D model file in the database. For the result of the semantic similarity between the first set and the second set, if the semantic similarity exceeds the similarity threshold, it is determined that the second set matches the first set, and the 3D asset corresponding to the second set is used as the matching result of this time.

[0086] Figure 3 The following shows a specific implementation process for implementing the retrieval of the target object by using the above retrieval algorithm in an embodiment of the present application, including the following steps:

[0087] Step S301: Input the target object to obtain the label. Obtain the target object to be queried from the query information input by the user, and generate the query label information.

[0088] Step S302: Construct the first set A based on the query label information.

[0089] Step S303: Read a row of records from the database in a preset order. The row of records includes the label information corresponding to a 3D model file already written in the database, and thus construct the second set B.

[0090] Step S304: Match the first set A and the second set B to determine whether the match is successful. In the process of matching the first set A and the second set B, instead of using a strong character matching algorithm, a more flexible semantic matching algorithm is adopted. Under the semantic matching algorithm, 'a rose' and 'rose' are regarded as basically equivalent, and there is also a certain similarity between 'rose' and 'flower', rather than simply judging based on the exact match of the character strings. The specific semantic matching algorithm can be implemented by various suitable NLP (Natural Language Processing) algorithms. For example, in this embodiment, the spaCy natural language processing library can be used to match each label in the first set A and the second set B. The following is the pseudo code for determining whether two label information label1 and label2 match successfully:

[0091] def semantic_similarity(label1,label2,threshold=0.8):

[0092] # Convert the label to a spaCy document object

[0093] doc1=nlp(label1)

[0094] doc2=nlp(label2)

[0095] # Calculate the similarity

[0096] similarity=doc1.similarity(doc2)

[0097] # Return whether the similarity is higher than the threshold

[0098] return similarity >= threshold

[0099] If the similarity is higher than the threshold, it means that the two labels match successfully.

[0100] Based on the results of whether each label in the first set A and the second set B matches successfully, the similarity between the first set A and the second set B can be calculated. The pseudo-code implementation is as follows:

[0101]

[0102] Through the above semantic matching algorithm, it can be efficiently and accurately determined whether the first set A and the second set B match successfully. If they match successfully, step S305 is executed; if they do not match successfully, step S303 is executed to read the next record from the database in the preset order.

[0103] Step S305, include the successfully matched second set B into the candidate list.

[0104] Step S306, determine whether the records in the database have been scanned completely. If so, step S307 is executed; if not, step S303 is executed to read the next record from the database in the preset order.

[0105] Step S307, sort the candidate list and return the matching result.

[0106] 3. Cross-modal feature retrieval.

[0107] When the user is having a conversation with the large language model and specifies that the large language model uses the cross-modal feature retrieval method for querying, then the input query information will also be subjected to cross-modal feature extraction to achieve vectorization. Then, the cross-modal feature vector of the query information obtained by vectorization is matched and queried with the cross-modal feature vectors already stored in the database of 3D model files. For example, the query information is "Find scenes related to the Chinese New Year, using the cross-modal feature retrieval mode". The large language model analyzes that this is in the cross-modal feature retrieval mode, and thus extracts the words related to the Chinese New Year and performs cross-modal feature extraction on these words. After achieving vectorization, it goes to the database to match the cross-modal feature vectors extracted from the 2D pictures of the 3D model files, thereby obtaining the corresponding matching results.

[0108] In addition, the above three search methods can also be combined arbitrarily. For example, when the user does not set the relevant content of the search method in the search sentence input, the full scene mode query can be used. At this time, the above search methods will be called one by one in the order of basic information search, target object search, and cross-modal feature search to complete the search, and then the intersection of each matching result can be taken as the final matching result.

[0109] Step S204, according to the matching results, determine the corresponding 3D model file as the search result of this search. In an actual scenario, if there are multiple matching results, the matching results can be sorted, and then according to the sorted matching results, the 3D model file corresponding to at least one matching result with the highest sorting is selected as the search result of this search, and returned to the user for display, so that the user can browse the search results in order of matching degree.

[0110] Since this solution can use 2D image retrieval technology to realize 3D model file retrieval when performing 3D model file retrieval, and can use appropriate retrieval methods to match in the database according to different query information input by users, it effectively improves the efficiency and accuracy of retrieval and realizes efficient retrieval of 3D assets.

[0111] Figure 4 The following is a flowchart of a method for implementing a 3D model file storage process using a solution provided by an embodiment of the present application. The high-value 3D model files to be stored are added to the database of 3D assets, and an index is created to provide a data basis for subsequent retrieval. The method mainly includes the following steps:

[0112] 1) Data input and preprocessing

[0113] This step mainly includes the input and preprocessing of the 3D model files and basic information to be stored. In actual scenarios, this can be achieved by popping up an interactive window for adding 3D model assets. In this interactive window, users can enter the following content:

[0114] 3D model asset files (users need to upload, and multiple files may be uploaded);

[0115] 3D model editing tools (such as 3ds Max, Blender, Cinema 4D, etc.);

[0116] 3D model description information (such as model source, application background, scene description, etc.);

[0117] 3D model size (will be automatically generated based on the uploaded file);

[0118] 3D model author (you can fill in more than one);

[0119] 3D model tags (users can manually specify tag information);

[0120] 3D model attached drawings (users can upload 2D pictures such as front views, top views, left and right views, etc., or automatically obtain them through preprocessing).

[0121] 2) Target object detection step

[0122] In the aforementioned interactive window interface, after the user selects the next step, they can enter the target detection step, which mainly performs target detection on the 2D pictures corresponding to the 3D model file, identifies the objects contained therein, and automatically assigns tag information.

[0123] 3) Vectorization processing step

[0124] This step mainly performs cross-modal feature extraction on the 2D pictures to obtain cross-modal feature vectors for representing the image features and text description features of the 2D pictures in the same vector space.

[0125] 4) Warehousing step

[0126] Write the 3D model file uploaded by the user, the input basic information, and the 2D pictures, tag information, cross-modal feature vectors, etc. obtained in the foregoing steps into the database of 3D assets.

[0127] Figure 5 Shows the processing flow when retrieving a 3D model file using a solution provided by an embodiment of the present application. When a user wants to retrieve a 3D model file, a large language model will pop up to communicate with the user for interaction and processing. The large language model will perform a classification query based on the query content input by the user, mainly classified into three retrieval methods: basic information retrieval, target object retrieval, and cross-modal feature retrieval. The above three retrieval methods can be executed separately or in combination. In an actual scenario, the implementation of this part can be considered to be achieved using an intelligent agent. The intelligent agent can analyze the intention of the user input, and then call different program modules to complete different retrieval tasks. For the matching results obtained by various retrieval methods, they can be sorted and then mapped to the corresponding 3D model file to display the retrieval results to the user.

[0128] Figure 6 Shows the overall architecture of a 3D asset database system implemented by an embodiment of the present application. The entire system adopts a loosely coupled architecture, and the functional modules use HTTP (Hypertext Transfer Protocol) messages for data interaction, mainly including the following functional modules:

[0129] The database module 610 includes a basic information database, a tag database, a vector database, a 2D graphics library, and a 3D model asset library. Among them, the basic information database can adopt a relational database to store the basic information of 3D model files; the tag database can adopt a key-value pair database to store the tag information obtained after target detection; the vector database is used to store the cross-modal feature vectors obtained by cross-modal feature extraction; the 2D graphics library and the 3D model asset library are respectively used to store 3D model files and 2D pictures, and can record the matching mapping relationship between 2D pictures and 3D model files.

[0130] The process control module 620 mainly guides the user to complete the process of storing and retrieving 3D model files. In the interactive process of storing and retrieving 3D model files, on the one hand, the currently widely used GUI can be adopted to guide the user to complete the input, and on the other hand, a large language model can be used for interaction, allowing the large language model to check the user's various input contents. By using the large language model for natural language-based interaction, the user can use a mode similar to natural language to store and retrieve 3D model files, improving the user's interaction experience.

[0131] The main functions of the process control module include:

[0132] 1) According to the user's input content, judge the user's intention, such as whether to upload a 3D model file or retrieve a 3D model file.

[0133] 2) If it is to upload a 3D model file, the user is required to supplement relevant data. For example, when the user wants to upload a 3D model file, the user is required to provide the 3D model file, input basic information (project description, author, production date, etc.), input relevant tag information, and then prompt the user to store it. After collecting the user's input relevant data, notify other functional modules to complete target detection, vectorization, storage, etc.

[0134] 3) If it is to retrieve a 3D model file, the user is required to input query information and determine the retrieval method. If it is the basic information retrieval method, the user is required to input retrieval conditions, and then notify the relevant basic information module to perform the retrieval. If it is the target object retrieval method, the user is required to input the main target object, and then notify the target object module to perform the retrieval. If it is the cross-modal feature retrieval method, the user is prompted to input relevant description information, and then notify the vectorization module to perform the retrieval. In addition, the retrieval also supports the combination of the above several methods, and the retrieval processes of each retrieval method are executed one by one.

[0135] 4) After the relevant functional modules return the retrieval results, they can be sorted according to certain rules.

[0136] The basic information module 630 mainly provides functions for storing and retrieving the basic information of 3D model assets. It mainly stores the basic information of 3D model assets, such as file names, generation dates, tags, authors, upload dates, etc., corresponding 2D files, etc., and supports retrieval based on the basic information.

[0137] The target object module 640 mainly analyzes the content of 2D pictures for object detection, detects relevant target objects from them, generates corresponding tag information and stores it in the database. When the user conducts a target object search, the tag information corresponding to the target object input by the user is compared with the content stored in the database, and the 3D model file with the most relevant tag information is retrieved.

[0138] The vectorization module 650 is used to map the image feature vectors and text feature vectors of 2D pictures into the same vectorized space, so as to realize cross-modal search of images and text. Its main functions include performing cross-modal feature extraction on 2D pictures, storing them after vectorization processing, and according to the query content input by the user, performing cross-modal feature extraction on various relevant texts of the query content, and retrieving them in the database after vectorization.

[0139] The interaction between the above-mentioned various functional modules can be carried out using the HTTP message mechanism, so as to adopt loose coupling. Several functional modules can be placed on a single device or on different devices. If the cluster deployment method is adopted, the following deployment methods can be used: 1. Control server: Deploy the process control module, basic information module, target object module, and vectorization module on a single server device. 2. Database server: Save the relevant information in the basic information database, tag database, and vector database on the same server device. 3. File server, store the data in the 3D model asset library and 2D graphics library in files and store them in a separate file server.

[0140] The main innovations of the above system are as follows:

[0141] 1. Convert 3D model files into 2D pictures, and use 2D picture retrieval technology to realize the retrieval of 3D model files, improving the retrieval efficiency and accuracy.

[0142] 2. Comprehensively utilize various 2D picture retrieval modes, such as vectorizing image features with the help of a multi-modal large model (LMM, Multimodal Large Models) to realize cross-modal retrieval of "text-to-image" and "image-to-text". For example, the ES (Elastic search) search engine can be used for tag search, and at the same time, file basic information search, such as file name, author, etc., is also supported, improving the convenience of interaction and facilitating users to complete retrieval in various ways.

[0143] 3. An algorithm based on tag information is formulated for the method of retrieving the target object, so that the target object search based on tag information can be realized without using a search engine, which is lighter than the ES search engine.

[0144] 4. The large language model is adopted to optimize and supplement the retrieval conditions of the user, so as to improve the user experience.

[0145] The embodiment of the present application also provides a computing device, which includes a memory for storing computer program instructions and a processor for executing the computer program instructions. When the computer program instructions are executed by the processor, the device is triggered to execute the foregoing 3D model file warehousing method or 3D model file retrieval method.

[0146] In particular, the method and / or embodiment in the embodiment of the present application can be implemented as a computer software program. For example, the embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium. The computer program includes program codes for executing the method shown in the flowchart. When the computer program is executed by the processing unit, the above functions defined in the method of the present application are executed.

[0147] It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device.

[0148] In this application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0149] The computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0150] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0151] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above computer-readable medium carries one or more computer program instructions, and the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of the foregoing multiple embodiments of the present application.

[0152] It should be noted that the present application can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0153] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claimed rights. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the apparatus claims can also be implemented by one unit or device through software or hardware. The terms first, second, etc. are used to denote names and do not denote any particular order.

Claims

1. A method for storing 3D model files, characterized in that, The warehousing method includes: Obtain the 3D model file to be warehoused and the corresponding basic information; Preprocess the 3D model file to obtain 2D images of at least one view of the 3D model file; Perform object detection on the 2D images, identify the objects included in the 2D images, and generate label information based on the objects included in the 2D images; Extract cross-modal features from the 2D images to obtain cross-modal feature vectors; Write the 3D model file, its corresponding 2D images, basic information, label information, and cross-modal feature vectors into the database.

2. The method according to claim 1, wherein Preprocessing the 3D model file to obtain 2D images of at least one view of the 3D model file includes: Preprocess the 3D model file to obtain 2D images of multiple key views of the 3D model file, and the key views include front view, side view, and top view.

3. The method according to claim 1, characterized in that, Extracting cross-modal features from the 2D images to obtain cross-modal feature vectors includes: Use the CLIP model to perform cross-modal feature extraction on the 2D images to obtain cross-modal feature vectors for representing the image features and text description features of the 2D images in the same vector space.

4. The method according to claim 1, wherein Before obtaining the 3D model file to be warehoused and the corresponding basic information, it further includes: Obtain the user input content; Judge the user's intention according to the user input content; If it is determined that the user's intention is to warehouse the 3D model file, prompt the user to input the 3D model file to be warehoused and the corresponding basic information.

5. A 3D model file retrieval method, characterized in that, The 3D model file is written into the database by using the method described in any one of claims 1 to 4, and the retrieval method includes: Obtain the query information input by the user; Determine the retrieval method according to the query information; According to the corresponding retrieval method, use the query information to match in the database to obtain a matching result; Determine the corresponding 3D model file according to the matching result as the retrieval result of this retrieval.

6. The method according to claim 5, wherein The retrieval method at least includes any one of the following: basic information retrieval, target object retrieval, cross-modal feature retrieval; According to the corresponding retrieval method, use the query information to match in the database to obtain a matching result, including: According to the corresponding retrieval method, use the query information to match with the basic information, label information, and / or cross-modal feature vectors in the database to obtain a matching result.

7. The method according to claim 6, wherein When the retrieval method is target object retrieval, according to the corresponding retrieval method, use the query information to match in the database to obtain a matching result, including: Construct a first set according to the query label of the target object in the query information; Use a semantic matching algorithm to semantically match the first set with a second set to determine the semantic similarity between the first set and the second set, where the second set is constructed from the label information corresponding to any 3D model file in the database; If the semantic similarity exceeds the similarity threshold, determine that the second set matches the first set, and use the 3D asset corresponding to the second set as the matching result of this match.

8. The method according to claim 5, wherein Before obtaining the query information input by the user, it further includes: Obtain the content input by the user; Judge the user's intention according to the content input by the user; If it is determined that the user's intention is to retrieve 3D model files, prompt the user to input query information.

9. The method according to claim 5, characterized in that Obtain the query information input by the user, including: Use a large language model to obtain the query information input by the user by having multiple rounds of conversations with the user.

10. The method according to claim 5, characterized in that According to the matching result, determine the corresponding 3D model file as the retrieval result of this retrieval, including: When there are multiple matching results, sort the matching results; According to the sorted matching results, screen at least one 3D model file corresponding to the top-ranked matching results as the retrieval result of this retrieval.

11. A computing device, wherein, The device includes a memory for storing computer program instructions and a processor for executing the computer program instructions. When the computer program instructions are executed by the processor, the device is triggered to execute the method described in any one of claims 1 to 10.

12. A computer-readable medium having computer program instructions stored thereon, the computer-readable instructions being executable by a processor to implement the method described in any one of claims 1 to 10.

Images