NFT cross-modal retrieval method, device and storage medium
Through the NFT cross-modal retrieval model and the two-stage search algorithm for confidence accumulation through dynamic component differential training, the problem of low NFT image retrieval accuracy is solved, and more refined and diversified search matching is achieved, which improves the search accuracy and efficiency.
Patent Information
- Application Number
- CN202410972733.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-07-19
AI Technical Summary
The existing NFT image retrieval methods have low accuracy and cannot match users' diversified search intentions. In addition, traditional methods can only achieve NFT project-level search and cannot conduct refined searches.
The NFT cross-modal search model based on dynamic component differential training is adopted. Through the confidence accumulation two-stage search algorithm, the semantic information input by the user is obtained and the most matching search results are output, thereby improving the search accuracy of the NFT image.
It improves the accuracy and efficiency of NFT image retrieval, can match users' search intentions more accurately, and enhances the accuracy and diversity of search results.
Smart Images

Figure CN118520153B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electronic digital data processing technology, and in particular to an NFT cross-modal retrieval method, device and storage medium. Background Art
[0002] With the rise of the concepts of "Metaverse" and "Web3.0", non-fungible tokens (NFTs) have begun to enter the public eye as a unique digital asset in a virtual space. In order to meet consumers' personalized needs for NFT products, NFT trading platforms need a search engine that recommends text / media files based on user input. However, due to the high similarity between NFT data and the huge NFT inventory, academia or industry still lacks efficient and accurate search methods or products.
[0003] With the continuous improvement of various infrastructures of the Metaverse, NFTs will be traded like ordinary commodities in the future. However, the current mainstream NFT secondary trading markets and NFT RPC service providers all use traditional text tag matching and other methods to implement NFT search functions. Only when the keywords entered by the user completely match the specific preset tags such as the NFT project name will valid data be returned, which also makes it difficult for search results to match the diverse search intentions of users. In addition, a single NFT project usually contains about 10,000 highly similar image tokens. Traditional search methods can only achieve search granularity at the NFT project level, and cannot achieve refined search, which makes it difficult for users to find their favorite NFT images through existing search technologies. Summary of the invention
[0004] The present invention provides an NFT cross-modal retrieval method, device and storage medium to solve the technical problem of low NFT image retrieval accuracy in the prior art.
[0005] The present invention provides a cross-modal retrieval method for non-fungible tokens (NFTs), comprising:
[0006] Obtaining semantic information for retrieval;
[0007] The semantic information is input into the NFT cross-modal retrieval model, and the retrieval results output by the NFT cross-modal retrieval model are obtained based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0008] In some embodiments, the training step of the NFT cross-modal retrieval model includes:
[0009] Construct NFT image-text pair dataset;
[0010] Obtain the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference;
[0011] The NFT cross-modal retrieval model is trained based on the component composition and the label information corresponding to the component composition.
[0012] In some embodiments, the obtaining of the component composition of each component in the NFT image-text pair data set and the label information corresponding to the component composition based on dynamic component difference includes:
[0013] Determine a first image based on the NFT image-text pair data set;
[0014] Perform bitwise operations on the first image and other images in the NFT image-text pair data set to obtain a mask image;
[0015] Performing image difference on the first image and the mask image to determine each component in the NFT image-text pair data set;
[0016] Based on the encoder, the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition are obtained.
[0017] In some embodiments, the method further comprises:
[0018] A feature index is constructed based on the component composition and the tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
[0019] In some embodiments, constructing the NFT image-text pair dataset includes:
[0020] Get NFT data;
[0021] Preprocessing the NFT data to obtain NFT image-text pair data;
[0022] The NFT image-text pair data is divided to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a validation set and a test set.
[0023] In some embodiments, the two-stage search algorithm based on confidence accumulation is used to obtain the retrieval results output by the NFT cross-modal retrieval model, including:
[0024] Obtaining a retrieval vector for the semantic information and obtaining a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder;
[0025] Calculate the target token probability based on the retrieval vector and the database vector;
[0026] Determining the project probability of each NFT project based on the target token probability in the same project;
[0027] A retrieval result is determined based on the target token probability in the same project and the project probability of the project.
[0028] The present invention also provides a non-fungible token NFT cross-modal retrieval device, comprising:
[0029] A first acquisition module, used to acquire semantic information for retrieval;
[0030] The second acquisition module is used to input the semantic information into the NFT cross-modal retrieval model, and obtain the retrieval results output by the NFT cross-modal retrieval model based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0031] In some embodiments, the non-fungible token NFT cross-modal retrieval device further includes a first training module, which is used for training the NFT cross-modal retrieval model, and the first training module includes:
[0032] The first construction submodule is used to construct an NFT image-text pair dataset;
[0033] A first acquisition submodule is used to acquire the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference;
[0034] The first training submodule is used to train the NFT cross-modal retrieval model based on the component composition and the label information corresponding to the component composition.
[0035] In some embodiments, the first acquisition submodule includes:
[0036] A first determining unit, configured to determine a first image based on the NFT image-text pair data set;
[0037] A first acquisition unit, configured to perform a bitwise operation on the first image and other images in the NFT image-text pair data set to acquire a mask image;
[0038] A second determination unit, configured to perform image difference on the first image and the mask image to determine components in the NFT image-text pair data set;
[0039] The second acquisition unit is used to obtain the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition based on the encoder.
[0040] In some embodiments, the first acquisition submodule further includes:
[0041] The first construction unit constructs a feature index based on the component composition and tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
[0042] In some embodiments, the first building block comprises:
[0043] A third acquisition unit, used to acquire NFT data;
[0044] A first processing unit, configured to pre-process the NFT data to obtain NFT image-text pair data;
[0045] The second processing unit is used to divide the NFT image-text pair data to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a verification set and a test set.
[0046] In some embodiments, the second acquisition module includes:
[0047] A second acquisition submodule, configured to acquire a retrieval vector of the semantic information and a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder;
[0048] A first determination submodule, configured to calculate a target token probability based on the search vector and the database vector;
[0049] A second determination submodule, for determining a project probability of each NFT project based on the target token probability in the same project;
[0050] The third determination submodule is used to determine the retrieval result based on the target token probability in the same project and the project probability of the project.
[0051] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any one of the NFT cross-modal retrieval methods described above.
[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the NFT cross-modal retrieval methods described above.
[0053] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the NFT cross-modal retrieval methods described above.
[0054] The NFT cross-modal retrieval method, device and storage medium provided by the present invention can obtain an NFT cross-modal retrieval model based on dynamic component differential training, and then use a confidence accumulation two-stage search algorithm based on the NFT cross-modal retrieval model to output the retrieval results that best match the semantic information input by the user, thereby improving the retrieval accuracy of the NFT image. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0056] Figure 1 It is a flowchart of the NFT cross-modal retrieval method provided by the present invention.
[0057] Figure 2 It is a schematic diagram showing that the search performance of the model provided by the present invention gradually decays as the search space increases.
[0058] Figure 3 It is a structural schematic diagram of the NFT cross-modal retrieval device provided by the present invention.
[0059] Figure 4 An example physical structure diagram of an electronic device is provided. DETAILED DESCRIPTION
[0060] In recent years, with the rapid expansion of multimodal data and the rapid development of deep learning technology, cross-modal retrieval technology has gradually received attention from the academic community. This technology aims to use data of one modality as a query to retrieve data of another type of modality related to it. For example, users can search for matching pictures or video content through text queries. This search method based on understanding of data content defines a new paradigm for data search. Unlike traditional search methods based on content label matching, cross-modal retrieval technology is highly robust to user input. The text / visual encoder in the fully trained AI model will encode semantically similar text / audio and video into a similar vector space, and obtain the target output by calculating its "distance" to the alternative features in the preset vector space in the database.
[0061] At present, the academic community has also proposed many methods to improve the search accuracy / speed of NFT images. Based on the representation type of information, cross-modal retrieval technology can be generally divided into real-valued representation learning and binary representation learning (also called cross-modal hashing).
[0062] The goal of real-valued representation learning is to encode various modal data into real feature vector representations, among which Canonical Correlation Analysis (CCA) is a classic unsupervised subspace learning method that aims to find linear relationships between different modal data in order to establish correlations between them. Recently, researchers have begun to explore the use of pre-trained multimodal models to further improve retrieval performance. Models such as OpenAI and Google have shown the effectiveness of capturing complex cross-modal relationships. These models are pre-trained on large-scale heterogeneous datasets. The core idea of these methods is to align and compare the feature representations of images and texts in a shared semantic space, thereby achieving more accurate cross-modal matching.
[0063] Binary representation learning methods aim to transform different data modalities into a common Hamming space, making cross-modal similarity searches faster. However, since the feature representation of objects is encoded as binary codes, retrieval accuracy is usually slightly reduced due to information compression. Cross-modal hashing methods aim to discover correlations between data of different modalities, thereby enabling cross-modal similarity searches. Among them, some papers extend spectral hashing from traditional unimodal settings to multimodal scenarios, where the hash function maps similar objects across views to similar hash codes, enabling cross-view similarity searches. Some people have also proposed a predictable dual-view hashing algorithm. They formulated an objective function that maintains the predictability of binary codes, and optimized the objective function by applying an iterative method based on block coordinate descent, so that the model can accurately encode information from different modalities into the corresponding semantic space.
[0064] Because NFT on-chain data is relatively difficult to obtain, the academic community does not have a data set related to NFT data, and cannot carry out research in related fields. The current mainstream multimodal model of images and texts is based on Internet image and text data training. The image and text data in the data set have relatively large differences in semantics and pixel areas. It is difficult for the model to learn the ability to model the subtle differences between images and texts. However, NFT is a typical type of high-similarity image and text data. The differences between tokens are often seen in several local pixel areas. Traditional multimodal models are difficult to apply directly. NFT retrieval requires a set of models that focus on subtle differences between images and texts, as well as corresponding result processing algorithms.
[0065] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0066] Figure 1is a flow chart of the NFT cross-modal retrieval method provided by the present invention, such as Figure 1 As shown, the method includes the following:
[0067] Step 101: Obtain semantic information for retrieval.
[0068] Step 102: input the semantic information into the NFT cross-modal retrieval model, and obtain the retrieval results output by the NFT cross-modal retrieval model based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0069] Specifically, in order to solve the above technical problems, we can first obtain the semantic information input by the user for retrieval, and then obtain the retrieval results of the semantic information through the trained NFT cross-modal retrieval model.
[0070] Furthermore, the NFT cross-modal retrieval model can also be called a dual-stream image-text model, which can utilize the NFT image-text pair dataset provided in the embodiment of the present application, and then be trained based on dynamic component differentiation.
[0071] The trained NFT cross-modal retrieval model can obtain retrieval results based on the input semantic information through a two-stage search algorithm of confidence accumulation.
[0072] The NFT cross-modal retrieval method provided by the present invention can obtain an NFT cross-modal retrieval model based on dynamic component differential training, and then use a confidence accumulation two-stage search algorithm based on the NFT cross-modal retrieval model to output the retrieval results that best match the semantic information input by the user, thereby improving the retrieval accuracy of the NFT image.
[0073] In some embodiments, constructing the NFT image-text pair dataset includes:
[0074] Get NFT data;
[0075] Preprocessing the NFT data to obtain NFT image-text pair data;
[0076] The NFT image-text pair data is divided to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a validation set and a test set.
[0077] Specifically, in order to train the NFT cross-modal retrieval model, the embodiment of the present application constructs an NFT image-text pair dataset, NFT1000.
[0078] First, you can download NFT data. According to statistics, among the many categories of NFT projects, Profile Picture (PFP) NFT projects account for more than 60% of the market transaction share. Ethereum blockchain has the most prosperous NFT ecosystem, so the top 1,500 PFP NFT projects with the highest transaction amount on Ethereum can be selected as the target of data collection.
[0079] Then, data screening can be performed. In the model training phase, the quality of data annotation plays a decisive role in the performance of the model. However, the quality of the attribute description files that come with NFT data varies. Therefore, in the data screening phase, NFT projects with completely repeated tokens, fewer than 500 NFT projects, and NFT projects with no obvious semantic information in the description files are removed. 1,000 projects can be selected from the 1,500 NFT projects downloaded in the first phase as the final data source.
[0080] Then perform data standardization. Traditional NFT data source files include JPG / JPEG, PNG, WebP, GIF, MP4 and other formats, but as the input of the model, the training data needs to have a unified data format. Here, all static files can be unified in format, such as converted to PNG, and dynamic media files can randomly select a frame as the representative frame of the source file. Finally, all files are unified in size, such as a width of 512 pixels and an adaptive height. After a series of standardized operations, the original 14TB of data can be compressed to 1.75TB, which greatly saves storage and computing resources.
[0081] Finally, the data is divided. The images in the same NFT project are mixed with a batch of fixed components. If the data is divided into training set, validation set, and test set within the project, it will inevitably cause the model to "peek" at the data in the test set during training, resulting in inaccurate model evaluation. For this reason, the data set can be divided based on the project as the basic unit, and the standards are: 800 projects and 6,178,429 image-text pairs in the training set; 50 projects and 383,916 image-text pairs in the validation set; 150 projects and 1,000,838 image-text pairs in the test set.
[0082] The NFT cross-modal retrieval method provided by the present invention constructs the industry's first and largest NFT image and text dataset, which can improve the efficiency of model training.
[0083] In some embodiments, the training step of the NFT cross-modal retrieval model includes:
[0084] Construct NFT image-text pair dataset;
[0085] Obtain the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference;
[0086] The NFT cross-modal retrieval model is trained based on the component composition and the label information corresponding to the component composition.
[0087] Specifically, a dynamic component difference framework can be built and model training can be performed. The similarity of NFT images is rooted in the fact that each image in the same project is a permutation and combination of specific layer components. A full permutation of a small number of components can produce a large number of different but similar NFT images. Images containing the same components are consistent at the pixel level, so image difference methods can be used to extract each component in the same project separately.
[0088] First, we can build an NFT image-text pair dataset according to the above method, and then perform image difference.
[0089] It is worth mentioning that due to the mutual occlusion between components, it is difficult to obtain a complete component by performing only one differential operation. Therefore, the image differential operation can be repeated multiple times to obtain multiple component fragments, and finally the component fragments are superimposed together to obtain a relatively complete and clean component itself.
[0090] Finally, based on the Transformer architecture, a text encoder is used to extract label information from the component name, and an image encoder is used to extract feature vectors from the component itself to obtain the component composition. Finally, the NFT cross-modal retrieval model is trained based on the image-text comparison of the component composition and the label information corresponding to the component composition to align the semantic information of the image and text.
[0091] The NFT cross-modal retrieval method provided by the present invention can obtain an NFT cross-modal retrieval model based on dynamic component differential training, and can use image differential methods to extract each component in the same project separately, which not only saves storage space for training data, but also improves the training efficiency of the model.
[0092] In some embodiments, the obtaining of the component composition of each component in the NFT image-text pair data set and the label information corresponding to the component composition based on dynamic component difference includes:
[0093] Determine a first image based on the NFT image-text pair data set;
[0094] Perform bitwise operations on the first image and other images in the NFT image-text pair data set to obtain a mask image;
[0095] Performing image difference on the first image and the mask image to determine each component in the NFT image-text pair data set;
[0096] Based on the encoder, the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition are obtained.
[0097] Specifically, we analyze the distribution of components in the metadata of the NFT project, find out the pictures containing the same component, and then randomly select a batch of pictures. We use the first picture as a template and perform bitwise operations on the remaining pictures to obtain a component mask. Based on the mask, we apply it to the template picture to obtain a component. Finally, we use a text encoder based on the Transformer architecture to extract label information from the component name, and use an image encoder to extract feature vectors from the component itself to obtain the component composition.
[0098] The NFT cross-modal retrieval method provided by the present invention can obtain an NFT cross-modal retrieval model based on dynamic component differential training, and can use image differential methods to extract each component in the same project separately, which not only saves storage space for training data, but also improves the training efficiency of the model.
[0099] In some embodiments, the method further comprises:
[0100] A feature index is constructed based on the component composition and the tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
[0101] Specifically, based on the trained NFT cross-modal retrieval model, image and text encoders are used to extract image and text embedding from the NFT data to be retrieved in the database, and vectors are aggregated based on NFT items to obtain a vector representation of the entire database.
[0102] It is worth mentioning that, in order to make the subsequent retrieval process faster, the present invention uses hierarchical quantitative indexing based on graph indexing technology to organize data features into a relatively ordered form in the vector space to support efficient nearest neighbor search.
[0103] Then, a data lookup table is established between the features and the real data, and a mapping relationship is established between the data feature index and the file index. In this way, after obtaining the specific feature index, the data to be retrieved can be quickly located within a time complexity of 0(1).
[0104] The NFT cross-modal retrieval method provided by the present invention can extract NFT image and text features and construct an index, which can make the subsequent retrieval process faster and further improve the retrieval efficiency.
[0105] In some embodiments, the two-stage search algorithm based on confidence accumulation is used to obtain the retrieval results output by the NFT cross-modal retrieval model, including:
[0106] Obtaining a retrieval vector for the semantic information and obtaining a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder;
[0107] Calculate the target token probability based on the retrieval vector and the database vector;
[0108] Determining the project probability of each NFT project based on the target token probability in the same project;
[0109] A retrieval result is determined based on the target token probability in the same project and the project probability of the project.
[0110] Specifically, Figure 2 is a schematic diagram showing that the model search performance provided by the present invention gradually decays as the search space increases. Figure 2 As shown in the figure, if a target token exists in a certain NFT project, there will be a large number of similar tokens in this NFT project. Experimental results show that the search precision of the same model will gradually decrease as the search space increases. In addition, single-stage retrieval will cause the sorted probability list to be occupied by a large number of similar tokens in the same project, which will make some NFT projects with potential correct answers lose the chance of being retrieved.
[0111] According to the above theory, a confidence accumulation two-stage search algorithm is designed.
[0112] First, confidence accumulation is performed. The inner product operation is performed on the input search vector and the vector database, and the cosine similarity is equivalent to the token probability to sort the results. The top X (topX) results are taken, where X is a set positive integer, to obtain the target token probability. The feature index is queried, and the target token probabilities from the same project are accumulated to obtain the query probability of each NFT project, and the projects are sorted according to this probability to obtain a list of candidate projects.
[0113] Then, the project is sorted. The query vector is used to perform another query operation on the feature index of the candidate project to obtain the candidate token as the query result of this project.
[0114] Finally, the results are aggregated. The project probability is used as a weight to multiply the target token probability in the project as the final probability of the candidate token, and the results are rearranged as the final retrieval result.
[0115] It is worth mentioning that the confidence accumulation two-stage search algorithm significantly reduces the search space during the second search, thereby improving the search accuracy. The search results take into account both the accuracy and diversity of the search results. Because of the use of graph indexes and graphics processing unit (GPU) acceleration, and the order of magnitude difference between the second search space and the first search space, the two-stage search scheme does not introduce significant latency changes.
[0116] The NFT cross-modal retrieval method provided by the present invention can first search the NFT project, and then conduct a secondary search to locate the token file within the project. Combined with the use of GPU parallel acceleration technology, it significantly reduces the search space while ensuring the search accuracy and diversity of search results.
[0117] Below are the experimental results of the NFT cross-modal retrieval method provided by the present invention.
[0118] The goal of the cross-modal retrieval task is to output the multimodal data that best matches the semantic information input by the user. Based on the dynamic component difference framework, the CLIP series models (CLIP, META-CLIP, EVA-CLIP) can be fine-tuned using the data from the training set in NFT1000, and then zeroshot reasoning is performed on 100 NFT projects in the test set, using top1-top20 as the main measurement indicators. The results are shown in Table 1. It can be seen that the retrieval accuracy of the fine-tuned models on NFT data is generally greatly improved, which verifies that the dynamic component difference training framework and the confidence accumulation search strategy proposed in this invention are indeed effective.
[0119] Table 1. Comparison of model search performance before and after using the fine-tuning framework for the CLIP-VIT-L model
[0120]
[0121] Different models have different characteristics. In order to verify the universality of the training framework, other models with different parameter values and encoder structures can be comprehensively tested, and all NFTs in the test set are integrated into the same search space. The resulting search space size is 1 million. Finally, zeroshot reasoning is performed based on the fine-tuned model. The results are shown in Table 2. From the results, it can be seen that the method proposed in the patent has universality at the model level.
[0122] Table 2. Performance improvement after fine-tuning different models
[0123]
[0124] In the era of big models + big models, more high-quality training data often indicates better model performance. In order to verify the data validity of NFT1000, NFT100_mini (80 projects randomly selected, 500 image-text pairs selected for each project, a total of 40,000 image-text pairs) and NFT1000_mini (800 projects, 1,000 image-text pairs randomly selected for each project, a total of 794,689 image-text pairs) can be designed in the same proportion. After fine-tuning the models of different sizes, they are uniformly reasoned on the NFT1000 test set, and the search accuracy is shown in Table 3. As can be seen from the data table, these pre-trained SOTA models can hardly fit the data from the NFT1000 dataset, indicating that the data distribution in NFT1000 is unique and novel. At the same time, it also verifies the general rule that the larger the amount of training data, the stronger the model performance.
[0125] Table 3. Model performance improvement under different data scales
[0126]
[0127] Figure 3 is a schematic diagram of the structure of the NFT cross-modal retrieval device provided by the present invention, such as Figure 3 As shown, the NFT cross-modal retrieval device provided by the present invention includes a first acquisition module 301 and a second acquisition module 302, wherein:
[0128] A first acquisition module 301, used to acquire semantic information for retrieval;
[0129] The second acquisition module 302 is used to input the semantic information into the NFT cross-modal retrieval model, and obtain the retrieval results output by the NFT cross-modal retrieval model based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0130] In some embodiments, the non-fungible token NFT cross-modal retrieval device further includes a first training module, which is used for training the NFT cross-modal retrieval model, and the first training module includes:
[0131] The first construction submodule is used to construct an NFT image-text pair dataset;
[0132] A first acquisition submodule is used to acquire the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference;
[0133] The first training submodule is used to train the NFT cross-modal retrieval model based on the component composition and the label information corresponding to the component composition.
[0134] In some embodiments, the first acquisition submodule includes:
[0135] A first determining unit, configured to determine a first image based on the NFT image-text pair data set;
[0136] A first acquisition unit, configured to perform a bitwise operation on the first image and other images in the NFT image-text pair data set to acquire a mask image;
[0137] A second determination unit, configured to perform image difference on the first image and the mask image to determine components in the NFT image-text pair data set;
[0138] The second acquisition unit is used to obtain the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition based on the encoder.
[0139] In some embodiments, the first acquisition submodule further includes:
[0140] The first construction unit constructs a feature index based on the component composition and tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
[0141] In some embodiments, the first building block comprises:
[0142] A third acquisition unit, used to acquire NFT data;
[0143] A first processing unit, configured to pre-process the NFT data to obtain NFT image-text pair data;
[0144] The second processing unit is used to divide the NFT image-text pair data to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a verification set and a test set.
[0145] In some embodiments, the second acquisition module includes:
[0146] A second acquisition submodule, configured to acquire a retrieval vector of the semantic information and a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder;
[0147] A first determination submodule, configured to calculate a target token probability based on the search vector and the database vector;
[0148] A second determination submodule, for determining a project probability of each NFT project based on the target token probability in the same project;
[0149] The third determination submodule is used to determine the retrieval result based on the target token probability in the same project and the project probability of the project.
[0150] Specifically, the above-mentioned NFT cross-modal retrieval device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned NFT cross-modal retrieval method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.
[0151] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the NFT cross-modal retrieval method, which includes:
[0152] Obtaining semantic information for retrieval;
[0153] The semantic information is input into the NFT cross-modal retrieval model, and the retrieval results output by the NFT cross-modal retrieval model are obtained based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0154] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0155] In some embodiments, the training step of the NFT cross-modal retrieval model includes:
[0156] Construct NFT image-text pair dataset;
[0157] Obtain the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference;
[0158] The NFT cross-modal retrieval model is trained based on the component composition and the label information corresponding to the component composition.
[0159] In some embodiments, the obtaining of the component composition of each component in the NFT image-text pair data set and the label information corresponding to the component composition based on dynamic component difference includes:
[0160] Determine a first image based on the NFT image-text pair data set;
[0161] Perform bitwise operations on the first image and other images in the NFT image-text pair data set to obtain a mask image;
[0162] Performing image difference on the first image and the mask image to determine each component in the NFT image-text pair data set;
[0163] Based on the encoder, the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition are obtained.
[0164] In some embodiments, the method further comprises:
[0165] A feature index is constructed based on the component composition and the tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
[0166] In some embodiments, constructing the NFT image-text pair dataset includes:
[0167] Get NFT data;
[0168] Preprocessing the NFT data to obtain NFT image-text pair data;
[0169] The NFT image-text pair data is divided to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a validation set and a test set.
[0170] In some embodiments, the two-stage search algorithm based on confidence accumulation is used to obtain the retrieval results output by the NFT cross-modal retrieval model, including:
[0171] Obtaining a retrieval vector for the semantic information and obtaining a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder;
[0172] Calculate the target token probability based on the retrieval vector and the database vector;
[0173] Determining the project probability of each NFT project based on the target token probability in the same project;
[0174] A retrieval result is determined based on the target token probability in the same project and the project probability of the project.
[0175] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the NFT cross-modal retrieval method provided by the above methods, the method including:
[0176] Obtaining semantic information for retrieval;
[0177] The semantic information is input into the NFT cross-modal retrieval model, and the retrieval results output by the NFT cross-modal retrieval model are obtained based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0178] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the NFT cross-modal retrieval method provided by the above methods, the method comprising:
[0179] Obtaining semantic information for retrieval;
[0180] The semantic information is input into the NFT cross-modal retrieval model, and the retrieval results output by the NFT cross-modal retrieval model are obtained based on a confidence accumulation two-stage search algorithm. The NFT cross-modal retrieval model is obtained based on dynamic component differential training.
[0181] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0182] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A cross-modal retrieval method for non-fungible tokens (NFTs), characterized in that: include: Obtaining semantic information for retrieval; Inputting the semantic information into an NFT cross-modal retrieval model, and obtaining a retrieval result output by the NFT cross-modal retrieval model based on a confidence accumulation two-stage search algorithm, wherein the NFT cross-modal retrieval model is obtained based on dynamic component differential training; The two-stage search algorithm based on confidence accumulation obtains the retrieval results output by the NFT cross-modal retrieval model, including: Obtaining a retrieval vector for the semantic information and obtaining a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder; Calculate the target token probability based on the retrieval vector and the database vector; Determining the project probability of each NFT project based on the target token probability in the same project; Determining a search result based on the target token probability in the same project and the project probability of the project; The training steps of the NFT cross-modal retrieval model include: Construct NFT image-text pair dataset; Obtain the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference; Training the NFT cross-modal retrieval model based on the component composition and label information corresponding to the component composition; The obtaining of the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference includes: Determine a first image based on the NFT image-text pair data set; Performing bitwise operations on the first image and other images in the NFT image-text pair data set to obtain a mask image; Performing image difference on the first image and the mask image to determine each component in the NFT image-text pair data set; Based on the encoder, the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition are obtained.
2. The cross-modal retrieval method for non-fungible tokens NFT according to claim 1 is characterized in that: The method further comprises: A feature index is constructed based on the component composition and the tag information corresponding to the component composition, and the feature index is used to locate the retrieved NFT data.
3. The cross-modal retrieval method for non-fungible tokens NFT according to claim 1 is characterized in that: The construction of the NFT image-text pair dataset includes: Get NFT data; Preprocessing the NFT data to obtain NFT image-text pair data; The NFT image-text pair data is divided to obtain the NFT image-text pair data set, and the NFT image-text pair data set includes a training set, a validation set and a test set.
4. A cross-modal retrieval device for non-fungible tokens (NFTs), characterized in that: include: A first acquisition module, used to acquire semantic information for retrieval; A second acquisition module is used to input the semantic information into an NFT cross-modal retrieval model, and obtain a retrieval result output by the NFT cross-modal retrieval model based on a confidence accumulation two-stage search algorithm, wherein the NFT cross-modal retrieval model is obtained based on dynamic component differential training; The two-stage search algorithm based on confidence accumulation obtains the retrieval results output by the NFT cross-modal retrieval model, including: Obtaining a retrieval vector of the semantic information and obtaining a database vector, wherein the database vector is a vector representation of an NFT database determined based on an encoder; Calculate the target token probability based on the retrieval vector and the database vector; Determining the project probability of each NFT project based on the target token probability in the same project; Determining a search result based on the target token probability in the same project and the project probability of the project; The training steps of the NFT cross-modal retrieval model include: Construct NFT image-text pair dataset; Obtain the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference; Training the NFT cross-modal retrieval model based on the component composition and label information corresponding to the component composition; The obtaining of the component composition of each component in the NFT image-text pair data set and label information corresponding to the component composition based on dynamic component difference includes: Determine a first image based on the NFT image-text pair data set; Performing bitwise operations on the first image and other images in the NFT image-text pair data set to obtain a mask image; Performing image difference on the first image and the mask image to determine each component in the NFT image-text pair data set; Based on the encoder, the component composition of each component in the NFT image and text data set and the label information corresponding to the component composition are obtained.
5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the cross-modal retrieval method of the non-fungible token NFT as described in any one of claims 1 to 3.
6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the cross-modal retrieval method of the non-fungible token NFT as described in any one of claims 1 to 3.
7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the cross-modal retrieval method of the non-fungible token NFT as described in any one of claims 1 to 3.
