Method for constructing text retrieval 3D model data set, computer program product and electronic device

Abstract

The present invention provides a method for constructing a text retrieval 3D model dataset, a computer program product and an electronic device. Among them, a method for constructing a text retrieval 3D model dataset includes: obtaining diverse data for constructing a 3D model dataset; clustering the acquired data to obtain a clustering result including text and 3D models, generating a query text according to the clustering result, constructing a pairing of the query text to the 3D model to construct a sub-dataset, and assembling the sub-dataset to obtain a text retrieval 3D model dataset. The purpose of fairly evaluating the accuracy of the retrieval algorithm is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of 3D retrieval technology, and in particular relates to a method for constructing a text retrieval 3D model data set, a computer program product and an electronic device. Background Art

[0002] With the emergence of 3D generation algorithms and applications, the number of stock 3D models is increasing, and correspondingly, the difficulty of finding a 3D model that meets the needs is also rising sharply. One of the simplest and most direct search methods is to search for 3D models that meet the text description through text.

[0003] Textanimar2023 proposed a dataset for retrieving 3D animal models from text. They collected 186 different animal models from planetzoo and constructed 525 different 3D models by merging the number of mesh faces, which ultimately constituted a total of 711 3D models. In terms of text, 150 English sentences were manually designed, each of which contains a clear description of the natural form of the animal and a contextual description that matches the posture required for the model to be used, in order to improve the accuracy and efficiency of 3D animal model retrieval. In the above method, the only types of 3D models to be retrieved are animals, which may introduce domain bias, resulting in unfair evaluation of the retrieval algorithm; and there are only a few hundred 3D models to be retrieved, and the variance of the retrieval algorithm evaluation index is large. The evaluation index also refers to the text to 2D image retrieval dataset, such as MSCOCOcaption. However, the original intention of constructing MSCOCO caption is to perform image caption, not retrieval, so the retrieval evaluation index designed to adapt to this dataset may not be reasonable. The existing datasets have a single category of data and contain a small number of 3D models. Currently, there is no suitable 3D model retrieval evaluation dataset that can fairly evaluate the accuracy of the retrieval algorithm, which hinders the construction of a text retrieval 3D model application. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a method for constructing a text retrieval 3D model dataset, a computer program product and an electronic device, which at least partially solve the problem in the prior art that the accuracy of the retrieval algorithm cannot be fairly evaluated.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for constructing a text retrieval 3D model dataset, comprising:

[0006] Obtain diverse data for building 3D model datasets;

[0007] The acquired data is clustered to obtain clustering results including text and 3D models. A query text is generated according to the clustering results, and a pairing of the query text to the 3D model is constructed to construct a sub-dataset. The text retrieval 3D model dataset is assembled based on the sub-datasets.

[0008] Optionally, clustering the acquired data to obtain a clustering result including text and 3D models includes:

[0009] Use an image encoder to extract features from the default front view rendering image of each 3D model to obtain image features;

[0010] First, a title is generated based on the text in the data, and then the features of the title are extracted to obtain text features.

[0011] Optionally, the acquired data is clustered to obtain clustering results including text and 3D models, including: clustering the extracted image feature representations and text feature representations based on a set clustering quantity experience, excluding clusters with less than a set number of elements, thereby obtaining clustering results whose similarity and quality meet set requirements, and the clustering results include clustering based on text clustering and clustering based on image clustering.

[0012] Optionally, after the step of obtaining the clustering result whose similarity and quality meet the set requirements, the following steps are included:

[0013] For each cluster based on text clustering, select the sample closest to the cluster center as the representative sample, obtain the sample id of the representative sample, then find the corresponding cluster based on image clustering according to the sample id, and supplement the 3D model based on image clustering to the cluster based on text clustering.

[0014] Optionally, generating a query text according to the clustering result includes:

[0015] Process the titles generated based on the text in the data to eliminate interference factors that hinder the search results;

[0016] Then, the titles that exclude the interference factors that hinder the retrieval results are defined in difficulty levels according to the subjects and attributes contained, thereby obtaining query texts of different complexities.

[0017] Optionally, the assembling of the text retrieval 3D model dataset based on the sub-datasets includes:

[0018] All selected sub-datasets are downsampled to ensure that the size of the sub-dataset conforms to a Gaussian distribution with a set mean, and the standard deviation of the sub-dataset is set to ensure that the minimum number of 3D models in the sub-dataset exceeds the set number of models.

[0019] Optionally, when the text retrieval 3D model dataset is used to evaluate the retrieval method, the evaluation indicators include: average precision, average normalized discounted cumulative gain, average first layer and average second layer; the average first layer and average second layer are used to evaluate the method relevance of the retrieval depth.

[0020] Optionally, the calculation formula of the average precision is as follows:

[0021]

[0022] Among them, mAP stands for average precision, Q stands for the total number of queries, and GT i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N i represents the size of the i-th sub-dataset, j represents each 3D model in the sub-dataset, rel k (i) represents the indicator function, which is equal to 1 if the k-th search term of the i-th query is relevant, otherwise it is 0;

[0023] The calculation formula of the average normalized discounted cumulative gain is as follows:

[0024]

[0025] Where mNDCG represents the mean normalized discounted cumulative gain, Q represents the total number of queries, and k i represents the size of the i-th sub-dataset, and NDCG represents the normalized discounted cumulative gain;

[0026] The calculation formulas for the average first layer and the average second layer are as follows:

[0027]

[0028] Among them, mFT represents the average first layer, mST represents the average second layer, Q represents the total number of queries, and GT represents the average first layer. i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N i represents the size of the sub-dataset for the i-th query, j represents each 3D model in the sub-dataset, rel j (i) represents the indicator function, which is equal to 1 if the j-th search term of the i-th query is relevant, and 0 otherwise.

[0029] In a second aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for constructing a text retrieval 3D model dataset as described in any one of the first aspects.

[0030] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0031] at least one processor; and,

[0032] a memory communicatively connected to the at least one processor; wherein,

[0033] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a text retrieval 3D model dataset as described in any one of the first aspects.

[0034] The present invention provides a method for constructing a text retrieval 3D model data set, a computer program product, and an electronic device. The method for constructing a text retrieval 3D model data set overcomes domain bias by acquiring diverse data, clusters text and 3D models in the data and then pairs them, generates a query text according to the clustering result, thereby constructing a sub-data set, and obtains a data set for retrieving 3D models. Since the data set contains query text and 3D models, domain bias is eliminated, and the number of 3D models is large, thereby achieving the purpose of fairly evaluating the accuracy of the retrieval algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0036] Figure 1 A schematic diagram of 3D model retrieval provided in an embodiment of the present disclosure;

[0037] Figure 2 A flowchart of a method for constructing a text retrieval 3D model dataset provided by an embodiment of the present disclosure;

[0038] Figure 3 A schematic diagram of filtering and cleaning the acquired data provided in an embodiment of the present disclosure;

[0039] Figure 4a A schematic diagram of the length distribution of words in a query text provided by an embodiment of the present disclosure;

[0040] Figure 4b A schematic diagram of the Gaussian distribution of the number of 3D models within a sub-data set provided in an embodiment of the present disclosure;

[0041] Figure 5 A functional block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0043] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0044] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The illustrations only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0047] like Figure 1 As shown in Figure 1, the retrieval evaluation index designed in the prior art to adapt to this dataset may not be reasonable. Only the retrieval that is completely consistent with the annotated title will be counted as a positive example. This is inconsistent with intuition, such as in Figure 1 In the search for "a cat", we hope to retrieve "a sleeping cat".

[0048] For ease of understanding, Figure 2As shown, this embodiment discloses a method for constructing a text retrieval 3D model dataset, including:

[0049] Obtain diverse data for building 3D model datasets;

[0050] Diversity data includes category diversity and style diversity. Category diversity includes cars, humans, animals, etc., and style diversity includes cartoon style, physically based rendering (PBR), realistic scanning, etc.

[0051] The acquired data is clustered to obtain clustering results including text and 3D models. A query text is generated according to the clustering results, and a pairing of the query text to the 3D model is constructed to construct a sub-dataset. The text retrieval 3D model dataset is assembled based on the sub-datasets.

[0052] Optionally, clustering the acquired data to obtain a clustering result including text and 3D models includes:

[0053] Use an image encoder to extract features from the default front view rendering image of each 3D model to obtain image features;

[0054] First, a title is generated based on the text in the data, and then the features of the title are extracted to obtain text features.

[0055] Optionally, the acquired data is clustered to obtain clustering results including text and 3D models, including: clustering the extracted image feature representations and text feature representations based on a set clustering quantity experience, excluding clusters with less than a set number of elements, thereby obtaining clustering results whose similarity and quality meet set requirements, and the clustering results include clustering based on text clustering and clustering based on image clustering.

[0056] Optionally, after the step of obtaining the clustering result whose similarity and quality meet the set requirements, the following steps are included:

[0057] For each cluster based on text clustering, select the sample closest to the cluster center as the representative sample, obtain the sample id of the representative sample, then find the corresponding cluster based on image clustering according to the sample id, and supplement the 3D model based on image clustering to the cluster based on text clustering.

[0058] Optionally, generating a query text according to the clustering result includes:

[0059] Process the titles generated based on the text in the data to eliminate interference factors that hinder the search results;

[0060] Then, the titles that exclude the interference factors that hinder the retrieval results are defined in difficulty levels according to the subjects and attributes contained, thereby obtaining query texts of different complexities.

[0061] Optionally, the assembling of the text retrieval 3D model dataset based on the sub-datasets includes:

[0062] All selected sub-datasets are downsampled to ensure that the size of the sub-dataset conforms to a Gaussian distribution with a set mean, and the standard deviation of the sub-dataset is set to ensure that the minimum number of 3D models in the sub-dataset exceeds the set number of models.

[0063] Optionally, when the text retrieval 3D model dataset is used to evaluate the retrieval method, the evaluation indicators include: average precision, average normalized discounted cumulative gain, average first layer and average second layer; the average first layer and average second layer are used to evaluate the method relevance of the retrieval depth.

[0064] Optionally, the calculation formula of the average precision is as follows:

[0065]

[0066] Among them, mAP stands for average precision, Q stands for the total number of queries, and GT i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N i represents the size of the i-th sub-dataset, j represents each 3D model in the sub-dataset, rel k (i) represents the indicator function, which is equal to 1 if the k-th search term of the i-th query is relevant, otherwise it is 0;

[0067] The calculation formula of the average normalized discounted cumulative gain is as follows:

[0068]

[0069] Where mNDCG represents the mean normalized discounted cumulative gain, Q represents the total number of queries, and k i represents the size of the i-th sub-dataset, and NDCG represents the normalized discounted cumulative gain;

[0070] The calculation formulas for the average first layer and the average second layer are as follows:

[0071]

[0072] Among them, mFT represents the average first layer, mST represents the average second layer, Q represents the total number of queries, and GT represents the average first layer. i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N irepresents the size of the sub-dataset for the i-th query, j represents each 3D model in the sub-dataset, rel j (i) represents the indicator function, which is equal to 1 if the j-th search term of the i-th query is relevant, and 0 otherwise.

[0073] In a specific application scenario, LD-T3D in this embodiment is a large-scale, multi-category text retrieval 3D model dataset, that is, obtained using the construction method disclosed in this embodiment. The construction method of the text retrieval 3D model dataset specifically includes:

[0074] 3D model preparation:

[0075] As a basic component of the dataset, the quality of 3D models greatly affects the quality of annotations and the reliability of the dataset. Therefore, the following criteria for selecting 3D data sources are designed: data volume, category diversity (e.g., cars, humans, animals), and style diversity (e.g., cartoon style, physically based rendering (PBR), realistic scans). ShapeNet has relatively low data quality and style diversity, while the large-scale OmniObject3D dataset has a small number of categories and lacks style diversity. In contrast, the Objaverse dataset derived from Sketchfab provides a wide range of 3D models in various styles and categories. In addition, it was observed that a considerable portion of the models in Objaverse XL were of low quality, so Objaverse 1.0 was selected as the data source for LD-T3D. Afterwards, the data was simply cleaned, retaining 490,000 3D models with high semantic quality and less ambiguity. Some filtered examples are shown in Figure 2. Figure 3 As shown, Figure 3 (a) is a 3D model with low semantic information, (b) is a 3D model of various types, (c) is a LiDAR hole 3D model, and (d) is a 3D model with transparency errors.

[0076] Labeling process:

[0077] Taking into account the annotation cost and quality, this example decides to create a federated dataset containing 1,000 sub-datasets. Each sub-dataset is generated based on clustering. One text query corresponds to one sub-dataset, and each sub-dataset contains an average of 100 target models. Therefore, the entire annotation process includes clustering, text query generation, sub-dataset assembly, and quality review, which will generate approximately 100,000 text-to-3D retrieval pairs. The detailed steps of annotation are as follows.

[0078] 3D model clustering: The purpose of clustering is to create relatively complex datasets by grouping related objects into individual sub-datasets while ensuring semantic diversity among the sub-datasets.

[0079] In the clustering process, the features of two modalities (i.e., text features and image features) are used, and the widely used K-means clustering is performed on these features. Specifically, for image features, the image encoder of CLIP is used to extract features from the default frontal view rendering image of each 3D model. For text features, the caption is first generated by Cap3D, and the features of the caption are extracted using E5.

[0080] Then, K-means clustering was performed on the feature representations obtained by the above method, where the number of clusters (i.e., K value) was empirically set to 5000. Subsequently, we observed that clusters with fewer elements usually showed lower similarity and quality, so we excluded clusters with less than 10 elements. Finally, 4848 clusters based on image clustering and 4933 clusters based on text clustering were obtained, each cluster usually containing about 90 3D models.

[0081] At the same time, we observed that the clusters obtained based on text clustering have higher intra-cluster correlation. We believe that the reason is that the generated captions comprehensively consider the multi-angle features of the 3D model, rather than the image of a single perspective. Therefore, the results based on text clustering are mainly used, and the results based on image clustering are used as a supplement.

[0082] First, for each cluster based on text clustering, select the sample closest to the cluster center as the representative and obtain the "model_id" of the representative sample. Then, we find the corresponding cluster based on image clustering according to "model_id" and add the 3D model based on image clustering to the cluster based on text clustering. In this way, each cluster contains about 180 samples on average, and the samples are more diverse to build more positive and negative pairs. Figure 1 As shown, firstly, the texts of three cats are clustered together to obtain a representative text of one cat, and then the 3D models of three cats are clustered together. Then, the text of one cat and the three 3D models are clustered together to obtain a clustering result, that is, the clustering structure used to construct the data subset includes one text and multiple 3D models.

[0083] Finally, we sorted the models in descending order using the number of likes provided by Objaverse, selected the publicly recognized high-quality models, and selected the top 1,000 clusters representing the most popular models as the sub-dataset.

[0084] Query text generation,For query text generation, of course, a piece of irrelevant text can be randomly selected as the query for each sub-dataset.,However, using irrelevant text queries usually leads to a lack of positive samples in the,sub-datasets, which impairs the evaluation accuracy of the retrieval method because it fails to fully,evaluate its performance. Therefore, it is proposed to use titles to save the annotation work of text queries.,Each 3D model has a corresponding title, and the title representing the 3D model is used as the,initial text query. However, the titles generated by Cap3D usually show a machine-generated style,such as "a 3D model" and "gray background". These meaningless,descriptions may disrupt the semantic representation of the text title and may hinder the,accurate evaluation of the retrieval method.

[0085] Therefore, all “3D models” and descriptions of the background were removed. At the same time, in order to cover more queries of different complexity, three difficulty levels (i.e., easy, medium, and hard) were defined. Specifically, “easy” refers to queries containing only one subject, “medium” refers to queries containing one subject and one attribute, and “hard” refers to queries containing one subject and multiple attributes. As the number of attributes increases, the semantic scope that the query can represent becomes narrower and narrower, which means that the retrieval difficulty increases. For example, the concept of “a cat” is broader than “a sleeping cat”.

[0086] Finally, the ratio of text queries of different complexity is set to approximately 1:1:1.

[0087] Sub-dataset assembly,After obtaining 3D models and text queries for each sub-dataset,,text-to-3D model pairings can be constructed.,Specifically, it is observed that the updated clusters contain about 180 models,on average, 892 models at most, and 26 models at least.,To avoid too many positive samples in a certain sub-dataset, for each,sub-dataset, half of the number of 3D models are randomly selected,to be added to the sub-dataset.

[0088] Subsequently, in order to avoid the negative impact of severe long-tail distribution in the sub-datasets and increase the diversity of different sub-datasets, all sub-datasets are downsampled to ensure that the size of the sub-datasets conforms to the Gaussian distribution with a mean of 100. For the standard deviation, in order to ensure that the minimum number of 3D models in the sub-datasets exceeds 30, the standard deviation is set to (100-30) / 3≈23 according to the three sigma rule.

[0089] Annotation,After building the initial dataset, the dataset was sent to a professional data annotation company for data annotation, and each text-to-3D model pairing was checked by three manual annotators. The quality review included the following rules:

[0090] To ensure that there must be at least one positive sample in each sub-dataset (guaranteed by the way the query text is constructed), manual annotators need to check whether the generated query matches the corresponding 3D model.

[0091] For other text-to-3D retrieval pairs in the sub-dataset, human annotators need to check whether the query text and the 3D model do not match.

[0092] Due to the existence of annotation deviation and ambiguity, the majority voting method is used to generate the final annotation results.

[0093] Statistics:

[0094] like Figure 4a and Figure 4b As shown in FIG. , some statistical data of the data set obtained using the construction method disclosed in this embodiment are shown. Figure 4a It can be found that the length of most text queries is within 10 English words, and most of them are within 5 words. Figure 4b It can be found that the number of 3D models in the sub-dataset is generally Gaussian distributed, with a mean of about 100.

[0095] The dataset is a federated collection of multiple sub-datasets, where a single query retrieves all objects within the same sub-dataset. Therefore, the evaluation metrics must be designed appropriately to ensure fair comparisons across these different sub-groups. The evaluation metrics used by the LD-T3D benchmark are provided as follows:

[0096] Average Mean Accuracy (mAP) is a key metric widely used in object detection and retrieval tasks. For text-based 3D model retrieval, the calculation formula of mAP is as follows:

[0097]

[0098] Where Q is the total number of queries, GT i represents the truth set of the i-th query, |GT i | represents its cardinality, N i is the size of the i-th sub-dataset, rel k (i) is an indicator function that is equal to 1 if the k-th retrieved term of the i-th query is relevant (matches the truth term), otherwise it is 0. For each 3D model mNDCG in the j-sub-dataset:

[0099] Normalized discounted cumulative gain (NDCG) is a performance metric mainly used for ranking learning in information retrieval and machine learning. It emphasizes placing relevant results at the top of the ranking list, and in applications, the performance distribution is evaluated based on different result relevance scores. This example defines the mean NDCG (mNDCG) as another evaluation metric based on the standard NDCG formula:

[0100]

[0101] Where Q is the total number of queries, k i is the size of the i-th sub-dataset.

[0102] mFT and mST:

[0103] In addition, many retrieval methods use Recall@K to measure the proportion of queries that retrieve correct items among the K points closest to the query in the embedding space. Recall@K is the evaluation standard recall. However, in LD-T3D, there are multiple correct items and the size of each sub-dataset is different. A fixed K is unfair to the larger subset. Therefore, two variants of Recall@K are added to the evaluation framework: the first layer (FT) and the second layer (ST). They respectively convert |GT i | and 2×|GT i | as K. Then, the mean FT (mFT) and mean ST (mST) are defined to evaluate the relevance of methods with different retrieval depths:

[0104]

[0105] Where Q is the total number of queries, N i Represents the size of the sub-dataset for the i-th query, rel j (i) is an indicator function that is equal to 1 if the j-th term of the ith query is relevant and 0 otherwise.

[0106] When evaluating the retrieval method, the data set constructed in this embodiment is used to evaluate the retrieval method, that is, the query text of the data set constructed in this embodiment is used to perform retrieval in the retrieval method. Different retrieval methods retrieve different 3D models. Some can retrieve all 3D models corresponding to the query text in the data set, while others can only retrieve some 3D models. For example, Figure 1 Corresponding models, some methods can retrieve three 3D models after inputting a cat, while some methods can only retrieve a 3D model of a cat, but cannot retrieve the first and third 3D models. Figure 1 The 3D model retrieved is compared with the 3D model corresponding to the query text in the dataset, and then the evaluation index is used to evaluate the comparison result.

[0107] This example proposes LD-T3D, the first large-scale benchmark for text-based 3D model retrieval, containing 100,000 text-to-3D model pairs, covering 89,000 unique 3D models and 1,000 descriptive text queries. The benchmark is uniquely designed to reflect real-world scenarios, allowing researchers to explore and address the challenges of text-based 3D model retrieval at an unprecedented scale and diversity.

[0108] Strict dataset collation and quality control, strict filtering rules were established to exclude models with poor rendering quality or ambiguity, and clustering was performed to create diverse sub-datasets of LD-T3D, while strict human quality review was performed during the annotation process.

[0109] A realistic and practical evaluation of existing methods is performed using a 1-to-N evaluation protocol and four evaluation metrics: mAP, mNDCG, mFT, and mST to simulate the real-world retrieval process. The evaluation framework is designed for search engine scenarios, where the goal is to retrieve all relevant 3D models for a given text query. Based on these metrics, an extensive analysis of the effectiveness of existing methods is performed.

[0110] The advantage of this embodiment is that it constructs a large-scale, multi-category text retrieval 3D model dataset - LD-T3D. Unlike the textanimar2023 dataset, which only contains animal categories and hundreds of 3D models, LD-T3D includes 89,000 multi-category 3D models and 1,000 text descriptions, each of which corresponds to multiple 3D model positive examples. The retrieval evaluation indicators are redesigned, and mAP, mNDCG, mFT and mST are used to more fairly evaluate the effect of the retrieval algorithm. As shown in Table 1, a benchmark based on six common algorithms is provided. These improvements enable the dataset constructed using the method of this embodiment to surpass the existing technology in scale and diversity, and improve the fairness and reliability of the retrieval algorithm evaluation through the redesigned evaluation indicators. Retrieval results of different methods. For CLIP, BLIP2, Openshape and Uni3D, the same text encoder of the corresponding method is used to extract the features of the text query, and the object features are obtained by fusing different 3D modal features. For fair comparison, for image features, the corresponding features are extracted from all 8 angles of the rendered image. For text features, the text features are extracted from the subtitles generated by Cap3D. The fusion method of different features from different modalities is based on addition operation.

[0111] Table 1. Benchmarks of six common algorithms

[0112]

[0113] The electronic device disclosed in the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0114] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the electronic device performs all or part of the steps of the method for constructing a text retrieval 3D model data set in the aforementioned embodiments of the present disclosure.

[0115] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0116] like Figure 5 The present invention provides a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, which is suitable for implementing the electronic device in the embodiment of the present invention. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0117] like Figure 5 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0118] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as magnetic tapes and hard disks; and communication devices. The communication device can allow the electronic device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 5An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0119] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, all or part of the steps of the method for constructing a text retrieval 3D model data set of an embodiment of the present disclosure are executed.

[0120] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0121] The computer program product disclosed in this embodiment can be stored in a computer-readable storage medium, which stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the method for constructing a text retrieval 3D model dataset in the above-mentioned embodiments of the present disclosure are executed.

[0122] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0123] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0124] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0125] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0126] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0127] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0128] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0129] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0130] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for constructing a text retrieval 3D model dataset, characterized in that: include: Obtain diverse data for building 3D model datasets; Clustering the acquired data to obtain clustering results including text and 3D models, generating query text according to the clustering results, constructing a pairing of the query text to the 3D model to construct a sub-dataset, and assembling the sub-datasets to obtain a text retrieval 3D model dataset; The clustering results include clustering based on text clustering and clustering based on image clustering; The clustering of the acquired data to obtain a clustering result including text and 3D model includes: Use an image encoder to extract features from the default front view rendering image of each 3D model to obtain image features; First, generate a title based on the text in the data, then extract the features of the title to obtain text features; Generating the query text according to the clustering result includes: The titles generated based on the text in the data are processed to eliminate interference factors that hinder the search results.

2. The method for constructing a text retrieval 3D model dataset according to claim 1, characterized in that: The clustering of the acquired data to obtain clustering results including text and 3D models includes: clustering the extracted image feature representations and text feature representations based on a set clustering quantity experience, excluding clusters with less than a set number of elements, thereby obtaining clustering results whose similarity and quality meet set requirements.

3. The method for constructing a text retrieval 3D model dataset according to claim 2, characterized in that: After the step of obtaining the clustering result whose similarity and quality meet the set requirements, the method further includes: For each cluster based on text clustering, select the sample closest to the cluster center as the representative sample, obtain the sample id of the representative sample, then find the corresponding cluster based on image clustering according to the sample id, and supplement the 3D model based on image clustering to the cluster based on text clustering.

4. The method for constructing a text retrieval 3D model dataset according to claim 1, characterized in that: Generating the query text according to the clustering result also includes: Then, the titles that exclude the interference factors that hinder the retrieval results are defined in difficulty levels according to the subjects and attributes contained, thereby obtaining query texts of different complexities.

5. The method for constructing a text retrieval 3D model dataset according to claim 1, characterized in that: The text retrieval 3D model dataset is obtained by assembling the sub-datasets, including: All selected sub-datasets are downsampled to ensure that the size of the sub-dataset conforms to a Gaussian distribution with a set mean, and the standard deviation of the sub-dataset is set to ensure that the minimum number of 3D models in the sub-dataset exceeds the set number of models.

6. The method for constructing a text retrieval 3D model dataset according to claim 1, characterized in that: When the text retrieval 3D model dataset is used to evaluate the retrieval method, the evaluation indicators include: average precision, average normalized discounted cumulative gain, average first layer and average second layer; the average first layer and average second layer are used to evaluate the method relevance of the retrieval depth.

7. The method for constructing a text retrieval 3D model dataset according to claim 6, characterized in that: The calculation formula of the average accuracy is as follows: Among them, mAP stands for average precision, Q stands for the total number of queries, and GT i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N i represents the size of the i-th sub-dataset, j represents each 3D model in the sub-dataset, rel k (i) represents the indicator function, which is equal to 1 if the k-th search term of the i-th query is relevant, otherwise it is 0; The calculation formula of the average normalized discounted cumulative gain is as follows: Where mNDCG represents the mean normalized discounted cumulative gain, Q represents the total number of queries, and k i represents the size of the i-th sub-dataset, and NDCG represents the normalized discounted cumulative gain; The calculation formulas for the average first layer and the average second layer are as follows: Among them, mFT represents the average first layer, mST represents the average second layer, Q represents the total number of queries, and GT represents the average first layer. i represents the truth set of the i-th query, |GT i | represents the cardinality of the truth set of the i-th query, N i represents the size of the sub-dataset for the i-th query, j represents each 3D model in the sub-dataset, rel j (i) represents the indicator function, which is equal to 1 if the j-th search term of the i-th query is relevant, and 0 otherwise.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for constructing a text retrieval 3D model dataset as described in any one of claims 1-7 is implemented.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a text retrieval 3D model dataset as described in any one of claims 1-7.

Images