A gpu-based multi-modal indexing integration method

Abstract

The application discloses a GPU-based multi-modal index integration method. The method comprises the following steps: acquiring multi-modal original data and extracting a feature vector; mapping the feature vector to a unified common semantic space; constructing a double-layer hybrid index structure on a GPU, wherein a first layer adopts IVF-PQ index for coarse screening, and a second layer adopts graph index for fine screening; after receiving a query request, performing two-stage parallel search on the GPU, first screening a candidate set through IVF-PQ index, and then fine screening a final result through graph index. The application solves the cross-modal semantic alignment problem through the unified semantic space, realizes high-precision and low-delay retrieval of massive multi-modal data by using GPU parallel computing and a double-layer index structure, and effectively overcomes the technical bottleneck that precision and efficiency are difficult to be considered in the prior art.

Description

Technical Field

[0001] This application relates to the field of computer information retrieval technology, and in particular to a multimodal data indexing and retrieval method that utilizes a graphics processor for efficient parallel computing. Background Technology

[0002] In existing technologies, single index structures (such as HNSW graphs or IVF-PQ) often struggle to achieve the optimal balance between retrieval accuracy, speed, and memory efficiency. While graph indexes offer high search accuracy, they are time-consuming to build massive datasets and require significant memory bandwidth. IVF-PQ indexes are memory efficient and fast to build, but their search accuracy is affected by quantization errors. Furthermore, directly applying these two structures to multimodal scenarios fails to fully consider the synergistic optimization between cross-modal semantic uniformity and GPU hardware characteristics. Therefore, an integrated indexing method is needed that combines the advantages of multiple indexes and is deeply optimized for GPU architecture and multimodal characteristics. Summary of the Invention

[0003] The purpose of this application is to provide a GPU-based multimodal index integration method. This method designs a novel two-layer hybrid index structure, combining the fast coarse screening capability of IVF-PQ with the high-precision fine sorting capability of graph index, so as to achieve ultra-fast retrieval of massive multimodal data while ensuring high retrieval accuracy.

[0004] To achieve the above objectives, this application provides a GPU-based multimodal index integration method, comprising the following steps: S1. Obtain raw data from different modalities, including text, images, and audio; S2. Using the pre-trained deep neural network model corresponding to each modality, extract the feature vectors of each modality data to form a multimodal feature vector set; S3. Transfer the multimodal feature vector set to the global memory of the GPU; S4. On the GPU, a unified, two-layer hybrid approximate nearest neighbor index structure is constructed for the multimodal feature vector set. The two-layer hybrid index structure consists of a coarse screening index in the first layer and a fine ranking index in the second layer. S5. Receive a query request, wherein the query request contains query data in at least one modality; S6. On the GPU, based on the dual-layer hybrid index structure, a two-stage parallel cross-modal similarity search is performed. First, the candidate set is quickly filtered out through the first-layer index, and then the final result is refined from the candidate set through the second-layer index. S7. Output the refined multimodal data results.

[0005] Optionally, in step S2, the pre-trained deep neural network model is: the pre-trained deep neural network model used to extract text modal feature vectors is the BERT-Large model; the model used to extract image modal feature vectors is the ViT-L / 16 model; and the model used to extract audio modal feature vectors is the VGGish model.

[0006] Optionally, in step S4, the construction of the two-level index structure includes: S41. The multimodal feature vector set is mapped to a unified public semantic space through a shared projection layer to obtain a unified feature vector set; S42. Based on the unified feature vector set, a first coarse screening index is constructed in parallel on the GPU. The first coarse screening index is an IVF-PQ index that combines inverted files and product quantization. S43. For each vector in the inverted list of the IVF-PQ index, construct a second-level fine-ranking index in parallel on the GPU. The second-level fine-ranking index is a graph structure index.

[0007] Optionally, in step S42, the IVF-PQ index construction includes: S421. Use the GPU-parallelized K-Means clustering algorithm to cluster the unified feature vector set, dividing all vectors into multiple cluster centers to form an inverted list; S422. For each vector in the inverted list, use a GPU-parallelized product quantization algorithm to compress and encode it, converting the high-dimensional vector into a short code.

[0008] Optionally, in step S421, the K-Means clustering algorithm divides the given dataset into K clusters, such that the sum of the squared distances from each vector to the center of its cluster is minimized, which can efficiently and in parallel construct the data block structure necessary for the inverted file index; the K-Means clustering algorithm is accelerated by GPU parallel computing, uses Euclidean distance as the similarity metric during the clustering process, and the number of clustering iterations does not exceed 50 to ensure that the cluster centers converge.

[0009] Optionally, in step S43, the graph structure index construction includes: S431. The first layer IVF-PQ index is used to perform a preliminary search on the multimodal feature vector set, and a candidate nearest neighbor set is generated for each data point; S432. On the GPU, an initial nearest neighbor graph is constructed in parallel based on the candidate nearest neighbor set; S433. The initial nearest neighbor graph is optimized using a graph optimization algorithm to improve retrieval efficiency and accuracy.

[0010] Optionally, in step S6, performing the two-stage parallel cross-modal similarity search includes: S61: Extract the query feature vector of the query data and project it onto the public semantic space to obtain a unified query vector; S62: Coarse screening stage: On the GPU, calculate the distance between the unified query vector and all cluster centers, and select the N nearest inverted lists as the candidate set; use PQ quantization distance calculation to quickly retrieve the Top-M preliminary nearest neighbor vectors from the N inverted lists to form a candidate vector set; S63: Fine ranking stage: On the GPU, the unified query vector and the candidate vector set are passed to the second-level graph structure index; multiple threads are started to perform precise nearest neighbor search in parallel on the local nearest neighbor graph corresponding to the candidate vector set, traversing and finding the final K nearest neighbor vectors; S64: Return the multimodal data corresponding to the K vectors as the search results.

[0011] Optionally, in steps S62 and S63, the calculations are all performed in parallel on the GPU. The coarse screening stage utilizes PQ asynchronous computation and memory access to hide data access latency; the fine ranking stage utilizes the GPU's shared memory to cache candidate vector sets and local graph structures to accelerate graph traversal and distance calculation.

[0012] Optionally, in the dual-layer hybrid index structure described in step S4, the first-layer index and the second-layer index are stored separately in the GPU memory. The quantized codebook and inverted list of the first-layer index are stored consecutively, and the graph structure of the second-layer index is stored in the form of an adjacency list or adjacency matrix.

[0013] This application proposes a GPU-based multimodal index integration method. By constructing a two-layer hybrid index structure under a unified semantic space and making full use of the massive parallel computing capabilities of GPUs, it solves the problems of difficulty in balancing multimodal retrieval accuracy and efficiency and high retrieval latency under massive data in the prior art. It achieves millisecond-level, high-precision cross-modal retrieval of hundreds of millions of multimodal data. Attached Figure Description

[0014] Figure 1 A flowchart illustrating a GPU-based multimodal index integration method provided in this application embodiment; Figure 2 This application provides a diagram of a two-layer hybrid index structure in a GPU-based multimodal index integration method. Figure 3 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application; The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] The main solution in this application embodiment is: A GPU-based multimodal index integration method, characterized in that the GPU-based multimodal index integration method includes the following steps: S1. Obtain raw data from different modalities, wherein the modalities include at least text, images, and audio; S2. Using the pre-trained deep neural network model corresponding to each modality, extract the feature vectors of each modality data to form a multimodal feature vector set; S3. Transfer the multimodal feature vector set to the global memory of the GPU; S4. On the GPU, a unified, two-layer hybrid approximate nearest neighbor index structure is constructed for the multimodal feature vector set. The two-layer hybrid index structure consists of a coarse screening index in the first layer and a fine ranking index in the second layer. S5. Receive a query request, wherein the query request contains query data in at least one modality; S6. On the GPU, based on the dual-layer hybrid index structure, a two-stage parallel cross-modal similarity search is performed. First, the candidate set is quickly filtered out through the first-layer index, and then the final result is refined from the candidate set through the second-layer index. S7. Output the refined multimodal data results.

[0016] Due to the technical bottlenecks faced by existing technologies in processing massive multimodal data, such as difficulty in balancing retrieval accuracy and efficiency, difficulty in cross-modal semantic alignment, and poor system scalability, this application provides a GPU-based multimodal index integration solution. By constructing a two-layer hybrid index structure under a unified semantic space, it reduces retrieval latency and index construction time, and improves GPU utilization.

[0017] Reference Figure 1 This application provides an embodiment of a GPU-based multimodal index integration method to construct a cross-modal retrieval system supporting billions of multimodal data (including images, text, and audio). The GPU-based multimodal index integration method includes: Step S1: In this embodiment, 1 billion (image, text, audio) paired data are collected as the base dataset for training and building the index, including 500 million image data, 300 million text data, and 200 million audio data.

[0018] Step S2: Standardize the collected data: Image data is uniformly adjusted to 512×512 resolution and converted to RGB format; Text data is segmented and encoded with a uniform length of 256 tokens; Audio data is uniformly resampled to 16kHz and truncated into 10-second segments. The NVIDIA A100 is a Tensor Core GPU based on the Ampere architecture, designed for AI, data analysis, and high-performance computing. In this embodiment, the GPU model is NVIDIA A100, and feature extraction is performed in parallel on the NVIDIA A100 GPU cluster. Vision Transformer (ViT) is a model that applies the Transformer architecture to computer vision tasks, effectively processing image data and performing well in multiple vision tasks. This embodiment uses the basic model ViT-B / 16 from the Vision Transformer (ViT) series to extract image features. The extraction process is as follows: Image data tensor quantization: Convert a single 512×512×3 RGB image into a tensor in the deep learning framework PyTorch; Normalized pixel values; Image segmentation: ViT-B / 16 segments the image into fixed-size patches, which are then converted into sequence vectors; Insert CLS label: At the very beginning of the 1024 Patch sequences, add a learnable 768-dimensional "CLS classification label" tensor with a shape of [1025, 768]. Positional encoding superposition: The pre-trained positional encoding tensor shape (tensor shape is [1025,768]) is added element by element to the tensor of the Patch sequence + CLS label to finally obtain the input sequence tensor [1025,768]. Batch processing adaptation: The tensor shape of the input sequence of N images is [N, 1025, 768]; Feature Encoding: The preprocessed [N, 1025, 768] sequence tensor is input into the ViT-B / 16 encoder module to complete the encoding of global semantic features. The encoder consists of 12 identical Transformer Encoder Blocks stacked together. Each layer contains two sub-layers: a Multi-Head Attention (MHA) layer and a feedforward neural network (MLP) layer. The MHA allows each patch vector to be associated with all other patch vectors to capture global semantics. The output of the MHA is fed into a two-layer fully connected MLP to complete the non-linear transformation of the features. The MHA+MLP operation is repeated in the 12-layer encoder, and each layer performs a deeper global encoding of the sequence features. Feature extraction and output: From the [N,1025,768] tensor output by the encoder, index the 0th position (the position corresponding to the CLS marker) to obtain the [N,1,768] tensor; remove redundant dimensions and compress [N,1,768] into [N,768], that is, each image corresponds to a 768-dimensional feature vector; BERT is a bidirectional model focused on semantic understanding. It learns general language knowledge through pre-training on large-scale unlabeled text and then fine-tunes it with a small amount of labeled data to quickly adapt to various NLP tasks. In this embodiment, the basic model BERT-base from the BERT series is used to extract text features. The feature extraction process is as follows: Word segmentation: Call the corresponding word segmenter pre-trained by BERT-base and load the vocabulary; for Chinese text, segment by character, and merge high-frequency words into a single token; for English text, segment words into sub-words; after word segmentation, automatically add [CLS] (classification tag) to the beginning of the sequence and [SEP] (separator tag) to the end. If it is a sentence pair (such as question-and-answer text), [SEP] needs to be added between the two sentences; generate the initial token sequence; Token encoding and length unification: Each token is mapped to a unique integer ID using a tokenizer vocabulary; if the original token sequence length (including [CLS] / [SEP]) > 256: truncate from the end to 256 tokens (keeping [CLS] at the beginning and [SEP] at the end, truncating redundant content in the middle); if the original token sequence length < 256: pad the end with [PAD] markers (ID=0) until the sequence length is 256; generate an AttentionMask; Batch tensor conversion: 300 million texts need to be processed in batches, converting the encoded results of individual texts into tensors of the deep learning framework PyTorch; Text semantic encoding: The core of BERT-base is to perform contextual semantic encoding on a sequence of 256 tokens through a 12-layer Transformer encoder, ultimately outputting 768-dimensional features. The input embedding layer converts the batch token ID tensor into a high-dimensional semantic vector, including word embedding, position embedding, and sentence type embedding fusion. Layer normalization is performed on the fused embedding vector, and then a Dropout layer is passed to output an embedding sequence tensor with the shape still [N, 256, 768]. The embedding sequence is input into a 12-layer stacked Transformer encoder. The Multi-Head Attention (MHA) layer in the Transformer encoder achieves "bidirectional global context association and captures semantic dependencies". The Feed Forward Network (FFN) layer in the Transformer encoder performs non-linear semantic transformation on the output of the MHA layer. MHA+FFN is repeatedly executed in the 12-layer encoder, and each layer strengthens the contextual semantic expression of the token. Finally, the output tensor of the 12-layer encoder is still [N, 256, 768], but the vector now contains complete contextual semantic information. Feature extraction and output: Index extraction, taking the vector at position 0 from the encoder output tensor to obtain a tensor of [N, 1, 768]; Dimension compression, removing redundant dimensions, converting it into a feature matrix of [N, 768] (N is the batch size, each row corresponds to 768-dimensional global features of one text). Wav2Vec 2.0 is a self-supervised speech representation learning model. Its core objective is to learn general and expressive speech features from a large amount of unlabeled raw audio data. In this embodiment, the Wav2Vec 2.0 model is used to extract audio features. The extraction process is as follows: Audio preprocessing: FFmpeg is an open-source multimedia framework that can process different types of audio and video files. FFmpeg is used to decode audio of different formats into raw PCM (Pulse-Code Modulation) data; the decoded audio is uniformly resampled to 16kHz; and the audio is uniformly processed into 10-second segments as needed. Feature extraction: The pre-trained Wav2Vec 2.0 model is used to convert the pre-processed audio waveform into a 512-dimensional feature vector; the original audio waveform is input into Wav2Vec 2.0, and Wav2Vec2Processor performs feature extraction and normalization; the input processed by the processor is passed to Wav2Vec2Model for model inference, and the feature map output by the last Transformer layer of the model is last_hidden_state(batch_size, sequence_length, hidden_size); the feature vectors of sequence_length time steps (each hidden_size dimension) are aggregated into a single, fixed-dimensional (512-dimensional) feature vector; A multimodal feature vector set is formed, including feature vectors of the corresponding modes extracted by the ViT-B / 16, BERT-base, and Wav2Vec 2.0 models.

[0019] Step S3: Perform data preprocessing on the multimodal feature vector set, integrating the feature vectors of different modalities into a regular continuous matrix, and uniformly converting them into floating-point types of the same bit length to ensure continuous alignment of memory layout; use CUDA API to allocate continuous space in GPU global memory that matches the data size; implement high-bandwidth asynchronous data transmission from CPU to GPU through fixed memory and cudaMemcpyAsync, allowing the CPU to continue executing other tasks during transmission; after the transmission is completed, perform stream synchronization to ensure that the data is complete and ready, and then start the GPU computing kernel.

[0020] Step S41: Design a 512-dimensional fully connected layer as a shared projection layer to project the multimodal feature vector set onto a 512-dimensional unified common semantic space, generating unified feature vectors. This projection layer is trained through multimodal contrastive learning (using the InfoNCE loss function). The training objective is to maximize the cosine similarity of paired image and text vectors in the common space and minimize the similarity of unpaired vectors. In this embodiment, the training process of multimodal contrastive learning includes model architecture, loss function design, and training process, as follows: Step S411, Model Architecture: The overall architecture adopts a two-stage "encoder-projector" architecture. In the encoder stage, a pre-trained single-modal model is used to extract high-dimensional features of images, text, and audio respectively. In the projector stage, modality-specific projection layers are used to project features of different dimensions onto a unified 512-dimensional space. Each modality is equipped with an independent but structurally similar projection layer. An L2 normalization layer is added after the output of the projection layer to ensure that the feature vectors of all modalities are located on the unit hypersphere, which facilitates the calculation of cosine similarity. Step S412: The InfoNCE (Information Noise Contrastive Estimation) loss function is used to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. For training data with a batch size of B, the InfoNCE loss is calculated as follows: Calculate the similarity matrix ,in ] indicates the first The characteristics of the sample and the first Cosine similarity between features of each sample; For each sample Positive samples are Negative samples are ; InfoNCE loss is defined as in, For the sample size, To query the encoded vector of sample i, The encoding vector of the positive sample corresponding to the query sample i; The encoding vector of the negative sample that does not correspond to the query sample i. Temperature coefficient; Multimodal contrast loss: Calculate all possible cross-modal contrast losses. Image-text contrast loss: uses image features as the anchor and text features as the target; Text-image contrast loss: uses text features as the anchor and image features as the target; Image-audio contrast loss: uses image features as the anchor and audio features as the target; Audio-image contrast loss: uses audio features as the anchor and image features as the target; Text-audio contrast loss: uses text features as the anchor and audio features as the target; Audio-text contrast loss: uses audio features as the anchor and text features as the target; Calculate the final loss, which is the average of the six losses mentioned above; Step S413, Training: The multimodal feature vector set is divided into a training set (80%), a validation set (10%), and a test set (10%). Use a pre-trained model to extract features from all data and store them as high-dimensional feature vectors; Training parameters were set as follows: Batch size: 512; Learning rate: initial learning rate set to 1e-4; damW optimizer (weight decay = 0.01); Temperature parameter (τ): 0.07; Number of training epochs: 150; Device: 8 NVIDIA A100 GPUs. Initialize model components and projection layers; iterate through the training data loader, acquiring one batch of trimodal data (images, text, audio) at a time; perform the following operations for each batch: Feature extraction uses pre-trained deep neural network models corresponding to different modalities to obtain features of different modalities. All feature extraction operations are performed in a gradient-free computation context (torch.no_grad()) to save memory and accelerate computation. Feature projection and normalization involve inputting different modal features into the corresponding projection layers to obtain the corresponding... A 512-dimensional intermediate feature vector is used to perform L2 normalization on all projected feature vectors to ensure that they lie on the unit hypersphere. Cross-modal loss calculation: All six possible cross-modal contrast losses were calculated, with each loss calculated using the InfoNCE loss function and the temperature parameter set to 0.07. Total loss calculation and backpropagation: Calculate the average of the six cross-modal losses as the total loss; clear the optimizer's gradient cache; perform backpropagation to calculate the gradient of the total loss relative to the projection layer parameters. Update the parameters of the projection layer using the optimizer; Loss accumulation and statistics: multiply the loss value of the current batch by the batch size and add it to the total loss; after all batches have been processed, calculate the average loss of the entire training set. Save the best model and compare the current validation loss with the historical best validation loss; if the current validation loss is lower, update the best validation loss and save the model parameters; the saved content includes the state dictionary of the three projection layers. Repeat the above steps until 150 training rounds are completed; after training, load the saved optimal model parameters for subsequent applications. Step S42: Construct the first layer (coarse screening layer) IVF-PQ index. The inverted file is responsible for "coarse screening". K-Means clustering is used to divide the vector space into blocks. During the search, only the nearest block is searched. Product quantization is responsible for "compression". First, the subspace is divided. Then, codebooks are built independently for each subspace. Finally, all vectors are quantized to represent high-dimensional vectors with a very small space, so that the vectors can be put into GPU memory. Step S421: Use the GpuIndexIVFPQ index in the GPU-accelerated Faiss library. Faiss (Facebook AISimilarity Search) is a high-efficiency similarity search library developed by Facebook, designed for large-scale high-dimensional vectors and providing GPU acceleration support; GpuIndexIVFPQ is a high-efficiency index structure in Faiss. In this embodiment, 10 million features are randomly sampled from 1 billion unified feature vectors as the training set, and K-Means clustering is performed on the GPU. K-Means is a classic unsupervised learning algorithm used to divide a dataset into K different clusters. During the clustering process, the number of clusters K is 10,000, the vector dimension d is 512, Euclidean distance is used as the similarity metric, and the number of clustering iterations does not exceed 50. Initialize cluster centers to ensure a reasonable initial center distribution; calculate the Euclidean distance from each sample to all cluster centers and assign the sample to the nearest cluster; calculate the average of all samples in each cluster and update the cluster centers; check if the change in cluster centers is less than a threshold or if the maximum number of iterations has been reached; generate 10,000 cluster centers (i.e., an inverted list) and use thousands of GPU cores to compute the distance between the vector and the center in parallel. Step S422: Product Quantization (PQ) is an efficient vector quantization technique designed to compress high-dimensional vectors into compact binary representations while preserving as much similarity information as possible between vectors. The following quantization steps are performed on each vector in the inverted list: Subvector extraction: The 512-dimensional vector space is divided into 8 subspaces, each with 64 dimensions; Parallel quantization: For each sub-vector, find the codeword with the closest Euclidean distance in the corresponding codebook; record the index (0-255) of the codeword in the codebook; the quantization of the 8 sub-vectors can be performed in parallel; PQ code generation: Combine 8 codeword indices together to form an 8-byte PQ code; each index is represented by a uint8 type (0-255); the original 512-dimensional floating-point vector (2KB) is compressed into an 8-byte PQ code.

[0021] Step S43: Construct the second-layer (fine-ranking layer) graph index: Hierarchical Navigable Small World (HNSW) is a graph-based approximate nearest neighbor (ANN) search algorithm. It achieves efficient similarity search by constructing a multi-layer graph structure and combining the characteristics of small-world networks. In this embodiment, HNSW is used as the fine-ranking layer index to provide accurate similarity ranking for the candidate results generated in the first stage (coarse-ranking layer). The process is as follows: Inverted list partitioning: The HNSW index is constructed based on 10,000 inverted lists generated by the first-stage K-Means clustering. Each inverted list corresponds to a cluster center, and each inverted list contains an average of 100,000 vectors. The vectors have been compressed into 8-byte PQ codes using product quantization technology. Local Graph Construction: Build an independent HNSW local graph for each inverted index, without sharing structure with other lists; randomly select a vector as the initial node of the graph, insert the remaining vectors sequentially, and search for new vectors. The nearest neighbor node, according to The parameters establish connections for new vectors, update the connections of related nodes, adjust the graph structure, and optimize the connection quality. For each local graph, set the following construction parameters: , The parameters control the number of neighbors explored during the mapping process; , The parameters control the average number of connections per node; the hierarchical decay factor is 0.75, controlling the decay rate of the number of nodes in each layer; search parameters... Used during the search phase, set to 50, to control search precision; Euclidean distance (L2 distance) is used as the similarity metric. Leveraging the parallel computing capabilities of GPUs, large-scale parallel graph building is achieved: the graph building task of 10,000 inverted lists is distributed to multiple stream processors of the GPU; each stream processor independently processes the graph building of one or more inverted lists; GPU resources are dynamically allocated to balance the load of each stream processor.

[0022] Step S5: Receive query requests; The system provides diverse query access interfaces, supporting multiple network protocols and data formats; The front end distributes query requests to different processing nodes through a load balancer to ensure system stability under high concurrency scenarios; Each query request contains complete metadata information for subsequent request routing, authentication, and priority processing. The system first automatically identifies the modality type of the query request, accurately determines the modality type through content sniffing and metadata analysis, parses the retrieval parameters in the query request, and uses contextual information to achieve personalized retrieval.

[0023] Step S61: Extract the query feature vector of the query data, and project the query feature vector onto the 512-dimensional unified public semantic space through a shared projection layer to obtain the unified query vector (the extraction process is the same as step S41).

[0024] Step S62, Coarse Screening Stage: On the GPU, the distance between the unified query vector and 10,000 cluster centers is calculated in parallel, and the N=16 nearest inverted lists are selected. In these 16 inverted lists (approximately 16 million vectors in total), the asymmetric distance calculation (ADC) of PQ is used. The GPU calculates the distance between each sub-vector of the unified query vector and the corresponding 256 codewords in the PQ codebook in parallel, forming 8 distance tables. Then, by looking up the tables and accumulating, the approximate distance between the unified query vector and these 16 million vectors is quickly obtained, and the TopM=512 candidate vectors and their original vector IDs are returned. Step S63, the fine-tuning stage execution process is as follows: Step S631, Candidate Vector Loading; Based on the 512 candidate vector IDs obtained in the coarse sorting stage, perform the following operations: Address mapping: Quickly locate the storage location of each candidate vector through the vector ID memory address mapping table; Batch loading: Loads the original 512-dimensional floating-point features of 512 candidate vectors from video memory in batch; Standardized format: Ensure all vector features are of type float32 and perform L2 normalization; Step S632, Shared Memory Data Organization; Organize the data into the GPU shared memory: Memory allocation: Allocate 64KB-128KB of shared memory for each thread block; Data partitioning: The 512 candidate vectors are divided into 16 blocks, each containing 64 vectors; Parallel loading: The query vector is copied to a fixed area of ​​shared memory; each thread block is responsible for loading one candidate vector block into shared memory; thread synchronization mechanisms are used to ensure data consistency. Step S633: Execute the HNSW graph search algorithm on the GPU: High-level navigation: Start from a random node at the highest level; quickly locate the vicinity of the target area by making a few jumps; gradually narrow the search range by moving down layer by layer; Local exact search: Perform a local search in the underlying graph; expand the neighbors of the current candidate node and update the nearest neighbor list; iterate and optimize until the most similar vector is found; Parallel search optimization: Multiple thread blocks process different candidate vector blocks in parallel; atomic operations are used to merge search results; a priority queue is used to maintain candidate nearest neighbors; Step S634, Sorting and Outputting Results: Similarity calculation: Calculate the exact Euclidean distance between the query vector and each candidate vector; Sorting and filtering: Sort by similarity in descending order, and select the TopK=100 most similar vectors; Results processing: Collect the original data IDs, modality types, and similarity scores of these vectors; generate the final list of search results.

[0025] Step S7: Return the original text descriptions corresponding to the Top-100 vectors to the user.

[0026] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores GPU-based multimodal indexing integration data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a GPU-based multimodal indexing integration method.

[0027] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0028] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0029] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0030] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0031] It should be noted that the engineering cost investment information and data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the client or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0032] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0033] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0034] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0035] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0036] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0037] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0038] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0039] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0040] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

[0041] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A GPU-based multimodal index integration method, characterized in that, The GPU-based multimodal index integration method includes the following steps: S1. Obtain raw data from different modalities, including text, images, and audio; S2. Using the pre-trained deep neural network model corresponding to each modality, extract the feature vectors of each modality data to form a multimodal feature vector set; S3. Transfer the multimodal feature vector set to the global memory of the GPU; S4. On the GPU, a two-layer hybrid index structure is constructed for the multimodal feature vector set. The two-layer hybrid index structure consists of a coarse screening index in the first layer and a fine ranking index in the second layer. S5. Receive a query request, wherein the query request contains query data in at least one modality; S6. On the GPU, based on the dual-layer hybrid index structure, a two-stage parallel cross-modal similarity search is performed. First, the candidate set is quickly filtered out through the first-layer index, and then the final result is refined from the candidate set through the second-layer index. S7. Output the refined multimodal data results.

2. The GPU-based multimodal index integration method as described in claim 1, characterized in that, In step S2, the pre-trained deep neural network model used to extract text modality feature vectors is the BERT-Large model; the model used to extract image modality feature vectors is the ViT-L / 16 model; and the model used to extract audio modality feature vectors is the VGGish model.

3. The GPU-based multimodal index integration method as described in claim 1, characterized in that, The specific steps for constructing the two-layer hybrid index structure in step S4 are as follows: S41. The multimodal feature vector set is mapped to a unified public semantic space through a shared projection layer to obtain a unified feature vector set; S42. Based on the unified feature vector set, a first coarse screening index is constructed in parallel on the GPU. The first coarse screening index is an IVF-PQ index that combines inverted files and product quantization. S43. For each vector in the inverted list of the IVF-PQ index, construct a second-level fine-ranking index in parallel on the GPU. The second-level fine-ranking index is a graph structure index.

4. The GPU-based multimodal index integration method as described in claim 3, characterized in that, The construction of the IVF-PQ index in step S42 further includes: S421. Use the GPU-parallelized K-Means clustering algorithm to cluster the unified feature vector set, dividing all vectors into multiple cluster centers to form an inverted list; S422. For each vector in the inverted list, use a GPU-parallelized product quantization algorithm to compress and encode it, converting the high-dimensional vector into a short code.

5. The GPU-based multimodal index integration method according to claim 4, characterized in that, In step S421, K-means clustering is accelerated by parallel computing on GPU. Euclidean distance is used as the similarity metric during the clustering process, and the number of clustering iterations does not exceed 50 to ensure that the cluster centers converge.

6. The GPU-based multimodal index integration method according to claim 3, characterized in that, In step S43, the fine-ranking index of the second layer is an index based on the nearest neighbor graph; Its construction process includes: S431. The first layer IVF-PQ index is used to perform a preliminary search on the multimodal feature vector set, and a candidate nearest neighbor set is generated for each data point; S432. On the GPU, an initial nearest neighbor graph is constructed in parallel based on the candidate nearest neighbor set; S433. The initial nearest neighbor graph is optimized using a graph optimization algorithm to improve retrieval efficiency and accuracy.

7. The GPU-based multimodal index integration method according to claim 3, characterized in that, The nearest neighbor graph constructed in step S432 is either a navigable small-world graph or a hierarchical navigable small-world graph.

8. The method according to claim 1, characterized in that, The two-stage parallel cross-modal similarity search performed in step S6 is specifically as follows: S61: Extract the query feature vector of the query data and project it onto the public semantic space to obtain a unified query vector; S62: Coarse screening stage: On the GPU, calculate the distance between the unified query vector and all cluster centers, and select the N nearest inverted lists as the candidate set; use PQ quantization distance calculation to quickly retrieve the Top-M preliminary nearest neighbor vectors from the N inverted lists to form a candidate vector set; S63: Fine ranking stage: On the GPU, the unified query vector and the candidate vector set are passed to the second-level graph structure index; multiple threads are started to perform precise nearest neighbor search in parallel on the local nearest neighbor graph corresponding to the candidate vector set, traversing and finding the final K nearest neighbor vectors; S64: Return the multimodal data corresponding to the K vectors as the search results.

9. The GPU-based multimodal index integration method according to claim 8, characterized in that, The calculations in steps S62 and S63 are performed in parallel on the GPU. The coarse screening stage utilizes PQ asynchronous computation and memory access to hide data access latency; the fine ranking stage utilizes the GPU's shared memory to cache candidate vector sets and local graph structures to accelerate graph traversal and distance calculation.

10. The GPU-based multimodal index integration method according to claim 1, characterized in that, The dual-layer hybrid index structure constructed in step S4 has its first-layer index and second-layer index stored separately in the GPU memory. The quantized codebook and inverted list of the first-layer index are stored consecutively, while the graph structure of the second-layer index is stored in the form of an adjacency list or adjacency matrix.

Images